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EXECUTIVE  SUMMARY 


This  project  covered  a  lot  of  ground.  The  working  premise  was  — 
and  remains  —  that  empirically  verifiable  findings  from  the 
cognitive  sciences  have  not  been  effectively  applied  in  systems 
design,  and  that  recent  advances  in  information  technology  have 
also  been  underexploited.  The  project  —  a  Phase  I  Small 
Business  Innovation  Research  (SBIR)  undertaking  —  demonstrates 
how  information  technology,  cognitive  science  and  a  specific  Air 
Force  domain  —  strategic  air  defense  —  can  be  merged  to  yield 
some  novel  system  concepts.  Several  prototypes  were  designed  and 
developed  that  demonstrate  how  strategic  air  defense  intelli¬ 
gence,  operations,  and  integrated  intelligence/operations 
processes  can  be  supported  with  interactive  computer-based 
systems.  The  overall  structure  of  the  project  appears  in  the 
following  figure. 


An  assessment  of  the  targe-  domain  was  undertaken.  Strategic  air 
defense  intelligence  analysis  and  production  is  located  within 
the  larger  and  more  generic  intelligence  analysis  and  production 
process.  A  set  of  tasks  and  sub-tasks  are  identified  and 
organized.  Strategic  air  defense  operations  are  examined  in  the 

r 

context  of  decision-making  and  planning.  The  domain  analysis 
also  suggested  a  range  of  possibilities  for  treating  the 
intelligence/operations  process  as  a  continuum.  In  spite  of  ^ 
doctrinal  precedent  to  the  contrary,  there  are  times  when  the 


/ _ 

intelligence  and.  operations  processes  should  be  intertwined.  v  Codes 


-  tLcd/or 

jOist  I  Special 


□  □ 


Project  Structure 


Intelligence  analysts  should  have  access  to  operational  plans, 
and  vice  versa. 

The  domain  analysis  was  followed  by  an  assessment  of  the  broad 
field  of  cognitive  science.  A  variety  of  findings  were 
identified,  findings  extremely  relevant  to  the  design  of  user- 
computer  interaction  routines  as  well  ac  ether  system  pioc°ices. 
Findings  were  identified  that  describe  how  humans  make  inferences 
and  decisions,  and  communicate  their  analytical  processes  and 
priorities.  Ws  noted,  for  example,  that  humans  make  inferences 
via  causal  schema  and  analogies,  and  decisions  based  upon 
multiple  criteria.  We  also  learned  that  a  variety  of  biases  and 
cognitive  predispositions  influence  the  inference-  and  decision¬ 
making  processes. 

An  assessment  of  advanced  information  technology  was  also 
undertaken.  This  assessment  concentrated  on  hypertext, 
multimedia,  interactive  graphics,  low-level  knowledge-based 
systems,  animation,  simulation  and  adaptive  user-computer 
interface  technology.  The  assessment  concluded  that  these 
technologies  are  ready  for  implementation,  cost-effective,  and 
naturally  synergistic  with  empirical  findings  from  cognitive 
science. 

A  set  of  interactive  "storyboard”  prototypes 

developed  to  demonstrate  how  cognitive  science  and  information 
technology  can  be  married  to  design  systems  compatible  with  the 
way  humans  structure  and  solve  problems.  These  prototypes  run 


V 


interactively  on  Apple  Macintosh  computers  and  suggest  how  air 
defense  intelligence,  operations,  and  intelligence/operations  can 
be  enhanced  via  the  use  of  interactive  computer-based  support 
systems.  Output  from  the  prototypes  appears  below. 

The  prototypes  suggest  clearly  that  the  working  premise  is  valid 
and  that  it  is  feasible  to  marry  information  technology  and 
cognitive  science  in  systems  that  will  —  at  least  hypothetically 
—  enhance  air  defense  intelligence,  operations  and  the  intel¬ 
ligence/operations  interface. 

We  also  demonstrated  how  intelligence  and  operations  can  be 
intertwined  to  yield  analyses  greater  than  the  sum  of  their 
individual  parts.  The  prototypes  suggest  that  the  conceptual 
and  functional  gaps  between  intelligence  and  operations  are 
minimal,  and  that  the  organizational  and  doctrinal  gaps  should 
be  closed. 

This  Phase  I  SBIR  project  has  demonstrated  that  "cognitive 
systems  engineering"  of  advanced  information  technology  is 
possible.  It  also  demonstrated  the  power  of  multidisciplinary 
research  and  development. 


vii 


viii 


AIR  DEFENSE  INTELLIGENCE 


PAUSE  SAVE  TO  OPERATIONS 


AIR  D 


CURRENT  situation  -  ' 


Qj 

CO 

cr 

to 

C_ 

CT 

c 

o 

QJ 

x: 

4-4 

cz 

CO 

o 

4-) 

to 

4-4 

to 

to 

QJ 

•O 

4-t 

to 

CO 

o 

o 

4-1 

c 

QJ 

to 

4-) 

to 

QJ 

t- 

jC 

4-* 

CL 

c 

o 

O 

W 

E 

CO 

3 

E 

QJ 

QJ 

4-J 

to 

’> 

>H 

o 

to 

to 

Qj 

QJ 

-C 

x: 

h- 

4-> 

4— 

O 

to 

Qj 

Q. 

to 

4.4 

E 

X 

CO 

!a 

CO 

to 

Q. 

CO 

c. 

u 

QJ 

to 

C.. 

QJ 

QJ 

t- 

X3 

o. 

E 

o 

to 

CL 

d 

to 

c 

o 

E 

.2 

4.4 

.2 

!E 

> 

»- 

to 

£Z 


(O 


O  Of  2 
Qj  ro 
OT  -O 


5 

a> 


OJ 

> 

o 

c: 

ro 


^  .— 
^  a 

CO  2 
CO  5 
QJ  ^ 


■D  _  . 

QJ  CJ  X 


QJ 

U 

QJ 


c 
o»  o 

c  z; 


QJ 
C 

CO  "to 
—  ^ 
o 


I 

o 

sr 

i  2 

x:  o 

&  ^ 

-  E 

^  o 
QJ  P 

■o  c: 

xi  ^ 

•—  (O 
O 


> 

CO 

x: 

I 

I 


ro 


S  p 

S  e 

_  CO 

u  *0 

9  to  c 
5:  c  o 

iA-»  ^ 


C 


E  i: 


O  4-J 

r:  to 


QJ  OJ 

to  t_ 
13  O 

QJ  E 


CO 

M 


^  o 

C  w 

c  to  f 
o  DJ 

—  t_ 

*=*  o 


QJ 

L. 

O 


QJ 
CO 
T3 
QJ  CO 
—  t_ 


to 


CO 

u 


c  rr 
o  ^ 


QJ 

to  .Si 

Q.  *j  _ 

to  (o 

■o  5  .s 

to  OT 

QJ  «-«  d 
—  C  X 
to  P  Ul 
C  O) 
E  3  x: 
E  u  ^ 


o 

to 

to 

QJ 

o 

o 


o 
c 

.X 
■O 

c 
to 

c  c 
o  o 


to  CO 


o  o 

W  4— 

c  c 


■O 

c. 

(O 

c: 

o 

Qj  '(ct 

^  s 

2  E 

■o  cn 

QJ  t_ 

b  ° 

Xj  qj 

o  o 

E  ^ 

4-1  — 

to  $ 

to  o 

—  c 

E 

£  -o 

to  ro 
3^  C 
(o  5 

$»  13 

E 

^  E 

.  L_ 

— 

CO 

t-  .E 

QJ 

1.”' 

rs 

4->  C 

I  to 

.9  E 

W  Z3 

o  x: 

Q. 

to 

T3  ^ 

C  QJ 

5 

2  z:  <D 

»»-  to  o 

O  g-  2 
QJ  E  o 
0.0.^ 
to  o  to 


XV 


E 

2 

E  E 
o 


(O 


(O 


^•2  E 

<D 

2  y  to 

is  « 

o  ^  * 

O'  2J 

tn  ^ 


•S  ^ 

w  "fi 


II 

O 

I  E 
fe 

<Q  p 
3  O 

22  to 
£  to 
> 
o  ^ 

(O  o 

■S  ® 
E  E 

i  - 

O  to 

.= 


o> 

s  d 

S  2 

>  "ti 
o  WJ 
C.  3 
Q,  ^ 

OT  W 
*-» 
E  c 
S  Q> 
•*-*  t 


a> 

4->  _ 

c  to 


to 

X3  TO 

i| 

(/> 

^  4-» 

^  b 
S  E 


•o  »“ 

S  •  o 
to  *0 

i  «  f 

55l 

p  « I 

a> 

—  to  ^ 
_,  p  a>  *j 

a>  c 

S2  CO 

5L5  8 

^  to 
t-  £  '3 
a>  ^  -c 

to  3  c 

3  T3  O 

to  t_  C_ 
o  OT  OT 

>  =»  3 
g>  a>  o» 

O.  4->  4-* 


moder  was  developed  from  the  system’s  embedded  models.  They  are 
presented  sIde-by-sIde  to  call  attention  to  the  differences  and  stimulate 
additional  user  analysis.  The  models  are  presented  as  "causar/"inf  luence" 
diagrams  of  the  situation  assessment  process,  since  analysts  tend  to 
diagnose  problems  causally.  The  user  then  selects  Analogies. 


Threat  for  additional  information. 
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Region  Operations  Control  Center  at  timenoorr  Ar-D  nedr  nnuiui  dye, 
Alaska.  Surveillance  has  detected  several  aircraft  in  the  zones  of 
coverage  that  surround  North  America.  The  target  ID  is  designated  as 
“pending,”  consistent  with  ROCC  operations.  The  user  clicks  on  Evaluate 
Air  Environment  to  get  more  information  about  the  situation. 


(O 


ra 

a; 

L- 

to 

O) 


S’  ^ 

ro  — 

to  <u 
^  -Q 

€  -o 
<1? 

ro  4— • 

—  u 

>  (U 

C  Q. 

o  X 

—  o 

4—) 

^  i:! 

u-  t- 

—  o 

•*-*  t_ 

S  c 
.  K> 


£  > 
O  O 
t- 

ro  Qj 


OT  "O 

u  to 
0.  ^ 
b  .S’ 

■O  — 
Q> 

O  — 

o  ro 
ko 

E 

g  I 

z  o 


XXIV 


The  system  determines  that  it  is  not  iiKeiy  that  it  is  commencai  or  civil¬ 
ian,  and  that  no  IFF  transmission  has  been  received.  It  concludes  that  the 
aircraft  is  unknown.  The  user  then  clicks  on  Red  Tracks. 
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on  Long-Term  Strike  &  Support  Aircraft  to  look  at  what  Intelligence 
believes  Is  likely.  The  system  presents  him  with  Its  best  estimate.  He 
then  clicks  on  Short-Term  Strike  &  Support  Aircraft.  A  variety  of 
"what-ir  activities  would  be  possible  via  such  Interaction. 
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1.0  INTRODUCTION 


1 . 1  Evolving  Information  Processing  Challenges 

Regardless  of  whether  the  task  is  to  detect  incoming  bombers  or 
determine  what  the  late  burning  lights  in  the  Kremlin  really 
mean,  today's  defense  analyst  is  buried  in  information  collected 
from  a  variety  of  sources.  The  strategic  intelligence  analyst 
has  too  much  data  and  too  little  time  to  make  too  many  judgments. 
Planners  must  contend  with  uncertainty,  intelligent  adversar¬ 
ies,  and  the  fog  of  war  all  seen  through  sensors  that  struggle  to 
correlate  strategic  and  tactical  phenomena. 

Over  the  past  twenty-five  years  we  have  placed  more  emphasis  upon 
the  collection  of  data  than  upon  its  correlation  and  analysis.  . 
Recently,  however,  decisions  have  been  made  to  optimise  the  uoe 
of  sensed  data  and  information.  The  operative  question  is  fast 
becoming:  "how  will  the  information  be  used?"  Systems  designers 

have  discovered  the  under-utilization  of  very,  very  expensive 
data,  and  have  begun  to  conceive  systems  that  distill  information 
^nto  highly  useful  content  and  form. 

Command,  control,  communications  and  intelligence  (C3I)  systems 
must  address  a  broad  range  of  problems.  Commanders  must  rely 
upon  intelligence  estimates  developed  from  incomplete  and 
uncertain  data  and  estimates  based  upon  less-than-perfect 
sampling  from  the  avalanche  of  data  to  which  analysts  have 
access.  Operations  analysts  must  interpret  the  intelligence 
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estimates  and  select  from  a  set  of  sanctioned  responses.  This 
template-like  approach  works  fine  until  completely  unanticipated 
events  occur  or  until  the  intelligence  estimate  becomes  ambiguous 
or  uninterpretable.  Operations  personnel  earn  their  pay  when 
they  have  to  induct  or  abduct;  deduction  is  relatively  painless. 

Too  much  information,  very  little  processing  time,  uncertainty, 
and  ambiguity  represent  but  a  few  of  the  problems  that  organic 
and  silicon  information  processors  face  every  day.  These 
problems  are  exacerbated  by  the  need  for  security,  survivability 
and  reconstitutability ,  among  other  requirements  unique  to  C3I 
systems  design,  development  and  deployment. 

Attention  is  also  now  turning  toward  the  organizations  that 
process  information,  and  the  humans  that  staff  them.  We  now  know 
that  human  information  processors  suflex  fi.uin  a  variety  of 
shortcomings,  including  biasing  and  anchoring.  We  know  that 
humans  find  it  difficult  to  make  inferences  from  ambiguous 
information  or  generate  comprehensive  sets  of  options.  We  know 
that  they  sometimes  reason  analogically,  though  the  process  by 
which  pertinent  analogies  are  selected  remains  largely 
confusing.  We  know  that  analysts  bring  mental  models  to  the 
problem-solving  process,  that  such  models  influence  the 
decision-making  process  enormously,  though  somewhat 
unpredictably. 

Finally,  we  know  that  the  processes  by  which  human  information 
processors  communicate  is  far  from  optimal.  Groups  are 
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notoriously  inefficient  and  members  find  it  difficult  to  share 
their  analytical  procedures.  This  latter  problem  is  especially 
vexing  when  procedures  assume  cooperation  and  understanding,  as 
when  intelligence  sends  estimates  to  operations. 

This  report  describes  research  that  addresses  a  subset  of  these 
information  processing  challenges.  The  focus  here  is  on  human 
information  processing  and  the  models ,  tools  and  techniques 
that  can  enhance  inference-making,  option  generation,  and  analyst 
communication. 

1 . 2  The  Information  Technology  Response 

If  we  were  to  assemble  a  room  full  of  computer  scientists 
and  ask  them  to  respond  to  the  above  challenges,  their 
responses  would  be  predictable.  The  problem  of  too  much 
information,  they  would  argue,  is  only  a  "throughput" 
problem:  all  we  need  are  faster  machines.  The  problem  of 

uncertainty  can  be  managed,  they  would  insist,  via  the 
application  of  fuzzy  sets,  Bayes'  Theorem,  Cohen's  model  of 
endorsement,  or  Dempster-Shafer  formulae.  In  spite  of 
occasional  breaches,  computer  security  is  well  within  reach, 
the  group  would  suggest,  as  is  our  redundancy  technology. 

While  few  share  all  of  the  above  optimism,  progress  in 
information  technology  has  in  fact  been  dramatic  over  the  past 
decade.  We  do  have  faster  machines.  Aspects  of  uncertainty  and 
ambiguity  can  be  addressed,  and  —  perhaps  most  importantly  — 


user  computer  interface  (UCI)  technology  has  evolved  to  the  point 
where  we  can  begin  to  blur  the  distinction  between  the  user  of 
the  system  and  the  system  itself.  Such  progress  results  in  new 
system  concepts  that  recognize  human  users  as  integral  parts  of 
the  total  system,  and,  therefore,  new  system  design  strategies. 

Progress  has  also  been  made  in  the  modeling  of  complex  behavioral 
and  analytical  phenomena,  modeling  that  can  be  displayed  and 
communicated  to  users  before,  during  and  after  the  computer-based 
problem-solving  process. 

There  is  a  vast  inventory  of  information  technology  at  our  dis¬ 
posal,  but  selecting  the  right  tool  from  the  inventory  can  be 
difficult.  Information  technology  represents  a  range  of 
solutions  that  can  only  be  successfully  applied  when  the  "match" 
is  right.  When  should  hypertext  be  used?  When  should  displays 
be  animated?  How  should  abstract  concepts  like  risk,  opportun¬ 
ity,  and  vulnerability  be  communicated  graphically?  These  and 
other  questions  can  only  be  answered  in  context.  Andriole 
{1989a)  describes  a  methodology  for  system  concept  design  that 
calls  for  the  identification  of  tasks,  profiles  of  the  system 
users,  and  organizational  requirements  prior  to  the  selection  of 
any  information  technology.  The  argument  calls  for  design 
prudence  and  pragmatism. 

The  procedures  for  matching  requirements  with  information 
technology  are  far  from  precise.  At  the  same  time,  there  is  a 
growing  body  of  empirical  evidence  that  informs  the  matching 
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process.  The  use  of  specific  interface  technology  (like  icons, 
mice,  windows  and  high  resolution  graphics)  can  now  be  justified 
given  user  experience  and  the  nature  of  the  tasks  to  be  performed 
on  the  system.  Smith  and  Mosier  (1984)  have  compiled  a  handbook 
of  sorts  to  help  with  the  design  of  effective  UCIs. 


s 
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lies  in  the  matching  of  specific  information  technology  to  user 


This  project  recognij.es  the  generic  power  of  current  and  emergin 


information  technolo 


tern  requirements.  An  additional  inventory  of  hypotheses 


and  findinqs  is  leveraged  here.  Our  research  suggests  that  the 


ower  of  information  technology  can  be  multiplied  when  combined 


with  findings  from  cognitive  science  and  emerging  principles  of 


Cognitive  systems  engineering  owes  its  origins  to  "low  level" 
human  factors  research.  The  industrial  and  systems  engineering 
communities,  with  a  great  deal  of  encouragement  from  psycholo¬ 
gists,  began  to  treat  human  factors  seriously  several  decades 
ago.  Early  research  was  oriented  to  ergonomic  concerns,  but  as 
the  years  passed  "high  level"  human  factors  foci  emerged.  High 
level  human  factors  research  inspires  the  design  and  development 
of  systems  compatible  with  the  way  humans  store  and  display  data 
and  information,  structure  problems,  and  generate  options.  High 
level  human  factors  speak  directly  to  the  synergism  between 
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cognitive  information  processing  and  system  capabilities. 
Cognitive  systems  engineering  provides  guidelines  for  the 
conversion  of  findings  in  the  cognitive  sciences  into  system 
concepts  compatible  with  their  human  components. 

The  principles  of  cognitive  systems  engineering  have  become 
important  over  the  past  few  years  because  of  the  growing  number 
of  inexplicable  system  failures.  In  fact,  the  "failure 
literature"  has  been  unable  to  pinpoint  recurring  problems  or 
attribute  failures  to  specific  flaws  in  the  conventional  design 
process  (Ligon,  1989).  It  is  clear,  however,  that  systems 
designed  without  reference  to  human  information  processing  and 
problem-solving  are  likely  to  alienate  their  users.  We  know  from 
past  painful  experience  that  if  users  have  to  re-learn  a  familiar 
problem-solving  process  to  accommodate  a  system's  quirks  then  the 
system  will  go  under-utilized  or  be  ignored  altogether  (Ramsey 
and  Atwood,  1979).  Designers  now  respect  the  need  for  "cognitive 
compatibility."  It  is  now  even  considered  cost-effective  to 
cognitively  engineer  simple  and  complex  systems. 

This  project  endorses  the  importance  of  cognitive  systems 
engineering  —  especially  when  coupled  with  current  and  emerging 
advanced  information  technology.  The  project  assumes  that 
complex  information  processing  problems  can  be  addressed  via  the 
cognitive  engineering  of  advanced  information  technology. 
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1 .4  Project  Overview 

The  working  premise  of  the  project  described  here  was  —  and 
remains  —  that; 


Recent  advances  in  cognitive  science,  systems 
engineering  and  advanced  information  technology  have 
provided  new  opportunities  for  C3I  systems  designers, 
opportunities  that  have,  by  and  large,  gone  unnoticed 
by  the  major  C3I  systems  houses.  It  is  now  possible 
to  dramatically  enhance  human  performance  via  the 
merger  of  the  principles  of  cognitive  systems  engi¬ 
neering  and  the  tools  and  techniques  of  advanced 
information  technology.  Cognitive  systems  engineering 
is  a  relatively  new  field  of  inquiry  that  calls  for 
the  design  and  development  of  computer-based  systems 
consistent  with  the  way  humans  store,  retrieve,  "dis¬ 
play,"  and  process  information.  The  advanced  informa¬ 
tion  technologies  available  to  us  now  include  hypertext, 
multimedia,  simulation,  animation,  cost-effective  color, 
interactive  graphics  and  deductive  knowledge-based  in¬ 
ference  models,  among  others.  The  principles  of  cogni¬ 
tive  systems  engineering  and  the  power  of  advanced  in¬ 
formation  technology  can  be  synthesized  to  yield  in¬ 
teractive  systems  that  can  greatly  enhance  human 
performance  in  C3I. 


1.4.1  Project  Tasks  -  The  research  described  here  was 
organized  around  five  tasks: 


•  Task  _1:  The  identification  and  analysis  of  a  suit¬ 
able  Air  Force  domain  to  demonstrate  the  cognitive 
engineering  of  advanced  information  technology; 

•  Task  An  assessment  of  the  major  progress  and 
findings  in  cognitive  science  and  cognitive  systems 
engineering,  especially  as  they  pertained  to  the  tar¬ 
get  Air  Force  domain; 

•  Task  2*  ^  description  and  assessment  of  the  major 

new  thrusts  in  applied  information  technology,  especially 
as  they  pertained  to  the  designated  domain  and  provided 
opportunities  for  application  of  validated  principles 
of  cognitive  engineering; 
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•  Task  4:  The  design  and  development  of  several 
Apple  Macintosh-based  working  "storyboard"  prototypes 
that  demonstrate  the  cognitive  engineering  of  advanced 
information  technology;  and 

•  Task  The  documentation  of  the  project  in  a 

final  technical  report  and  a  set  of  working  demonstra¬ 
tion  prototypes. 


1.4. 1.1  The  Target  Domain;  Air  Defense  -  Air 
defense  intelligence  and  operations  provides  a  fertile  ground  for 
analysis.  On  the  inference-making  (intelligence)  side  are  the 
classic  monitoring,  warning  and  estimation  tasks  that  define  so 
much  of  the  intelligence  process.  On  the  operations  side  we  find 
tasks  connected  with  situation  assessment  (informed  by  intelli¬ 
gence),  option  generation,  evaluation  and  selection.  We  began 
with  a  set  of  tasks  that  were  refined  in  an  air  defense  require¬ 
ments  analysis.  Some  of  these  tasks  included: 

INTELLIGENCE  TASKS 

e  Assessment  of  events  and  conditions; 

e  Inferences  about  likely  outcomes; 

e  Identification  of  high  probability  crisis  situations; 

e  Mechanisms  and  procedures  for  discriminating  between 
"signals"  and  "noise"; 

e  Qualitative  and  quantitative  indicator  development  and 
synthesis; 

e  Procedures  for  mapping  relationships  among  events;  and 

•  Extending  warning  lead  time  via  backchaining  "causal" 
data,  indicators,  and  activities,  among  others. 
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OPERATIONS  TASKS 


•  Option  generation; 

•  Assessment  of  operational  constraints; 

•  Cost/benefit  options  analysis; 

•  Search  for  historical  precedents; 

•  Sensitivity  ("what  -  if")  analysis; 

•  Option  outcome  assessment;  and 

•  Option  rank-ordering,  among  others. 

These  tasks  were  interpreted  specifically  with  reference  to  air 
defense  intelligence  and  operations  (see  Section  2.0  of  this 
report).  In  addition  to  the  requirements  analysis  of  air  defense 
—  as  it  is  conventionally  understood  and  practiced  —  we 
injected  yet  another  project  task:  analysis  of  the  interface 
between  intelligence  and  operations. 

This  added  dimension  of  the  project  required  us  to  first  analyze 
the  processes  by  which  intelligence  estimates  and  operational 
responses  were  formulated,  independent  of  one  another  (as  is 
often  the  case)  and  then  as  a  continuum,  where  intelligence 
becomes  the  front-end  to  a  process  that  iterates  upon  options 
(and  refined  intelligence  estimates)  until  an  appropriate 
response  is  identified.  Conceptually,  the  continuum  metaphor  is 
accurate,  but  in  practice  the  intelligence/operations  process  is 
much  more  stochastic  and,  at  times,  even  totally  disconnected. 
Intelligence  is  often  treated  by  operations  personnel  as  static; 
proposed  here  is  treatment  of  the  variable  as  dynamic.  More  to 
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the  point,  intelligence  and  operations  analyses  should  be  shared 
by  all  parties,  since  insight  into  the  production  or  consumption 
of  information  and  knowledge  will  widen  the  analytical  horizons 
of  both  sides.  Subsequent  project  research  was  thus  conducted 
around  discrete  and  continuous  assumptions  about  the  Air  Defense 
intelligence  and  operations  processes. 

1.4. 1.2  Cognitive  Science  and  Systems  Engineering 
-  Recently  a  number  of  psychologists,  information  scientists,  and 
systems  engineers  have  joined  forces  to  create  a  new  field  of 
inquiry  currently  known  as  cognitive  systems  engineering.  The 
key  word  in  the  phrase  is  "engineering,"  because  it  reflects  an 
intention  to  apply  what  we  know  about  human  information 
processing  to  larger  systems  engineering  efforts.  Had  "cognitive 
engineers"  been  around  (in  name,  at  least)  in  the  1970s,  they 
would  have  invented  windows,  the  mouse,  navigational  aids, 
spatial  data  base  management,  direct  manipulation  interfaces,  and 
interactive  graphics. 

During  the  course  of  the  research  described  here,  we  attempted  to 
take  what  is  "known”  about  cognitive  information  processing  and 
overlay  it  onto  air  defense  intelligence  and  operations 
requirements.  An  assessment  was  undertaken  to  distill  key 
findings  from  the  cognitive  science  literature,  findings  that 
could  be  leveraged  against  our  domain  requirements  (see  Section 
3.0).  The  assessment  permitted  us  to  isolate  a  set  of 
hypotheses,  tools  and  models  that  could  —  when  coupled  with 
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advanced  information  technology  —  be  used  to  enhance  air  defense 
intelligence,  operations,  and  the  intelligence/operations 
interface . 


1.4. 1.3  Advanced  Information  Technology  - 
Substantive  requirements  in  air  defense  and  findings  in  cognitive 
science  about  inference-making  and  option  selection  beg  the 
technology  question.  How  can  advanced  information  technology  be 
leveraged  against  requirements  and  what  we  know  about  cognitive 
information  processing?  Can  information  technology  be  cogni¬ 
tively  engineered?  The  prototypes  described  in  Section  5.0 
suggest  that  it  can,  though  a  series  of  experiments  designed  to 
measure  their  contribution  to  human  performance  should  now  be 
conducted . 

For  the  purposes  of  this  project  information  technology  was 
defined  very  broadly.  It  includes  all  of  the  tools,  techniques, 
models  and  methods  used  to  represent  complex  phenomena  with  a 
high  data,  information  or  knowledge  content.  Obviously,  the 
revolution  in  affordable  distributed  computing  has  played  a  part 
in  the  evolution  of  information  technology;  today  it  is 
impossible  to  address  the  field  independent  of  hardware  and 
software  correlates. 

The  emphasis  here  is  on  advanced  information  technology, 
technology  that  is  new  yet  cost-effective  enough  to  be  considered 
viable  for  real  applications.  The  technology  solutions  that  fall 


11 


into  this  category  today  (that,  incidentally,  would  not  have  been 
admitted  just  five  years  ago)  include: 


•  Hypertext; 

•  Multimedia; 

•  Animation; 

•  Real-time  simulation; 

•  Interactive  color  graphics; 

•  Deductive  inferential  knowledge  bases;  and 

•  Adaptive  and  direct  manipulation  interfaces,  among 
others. 

Our  research  suggests  that  (a)  enormous  leverage  can  be  gained 
via  the  cognitive  engineering  of  these  technologies  and  (b)  by 
and  large  the  technologies  have  been  grossly  under-exploited  by 
modern  systems  engineers  (see  Section  4.0),  ^ 

1.4. 1.4  Demonstration  Prototypes  -  Section  5.0 
describes  the  prototypes  that  we  developed  to  demonstrate  how 
advanced  information  technology  can  be  cognitively  engineered  to 
enhance  air  defense  intelligence,  operations  and  the  intelli¬ 
gence/operations  interface.  These  prototypes  run  interactively 
on  Apple  Macintosh  (Plus,  SE  and  II)  series  computers.  They  are 
designed  to  illustrate  how  systems  compatible  with  human  infor¬ 
mation  processing  can  be  designed  by  exploiting  findings  in 
cognitive  science  and  recent  progress  in  information  technology. 
Figure  1.1  locates  the  prototypes  within  the  larger  project 
organization. 
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Figure  1:  Project  Organization 


1.4. 1.5  Project  Documentation  -  This  report 
represents  only  part  of  the  project's  documentation.  In  addition 
to  tne  final  technical  report,  we  produced  a  set  of  working 
prototypes;  we  delivered  these  prototypes  along  with  supporting 
files,  run  time  versions,  and  necessary  applications  software. 

1.4.2  Project  Summary  -  The  project  is  chronicled  in  this 
report  and  the  prototypes;  the  Executive  Summary  above  and 
Section  7.0  distill  our  iindings  into  a  few  pages.  At  the  very 
least,  we  have  demonstrated  that  leverage  can  be  gained  from  the 
marriage  of  cognitive  science  and  advanced  information  tech¬ 
nology.  The  evidence  has  yet  to  be  empirically  verified,  but 
the  means  for  testing  the  efficacy  of  the  marriage  are  well 
within  reach.  The  prototypes  demonstrate  what  is  technically 
feasible  and  analytically  justifiable  today;  they  also  suggest 
a  new  research  emphasis  for  the  future. 

The  call  for  experimentation  is  not  mere  lip-service  to  the 
process  by  which  hypotheses  are  tested  and  confirmed,  rejected, 
or  modified.  Too  many  of  our  advanced  systems  concepts  —  while 
intuitively  exciting  —  have  never  been  empirically  tested. 

While  our  prototypes  may  or  may  not  represent  exciting,  intui¬ 
tively  appealing  system  concepts,  before  they  can  be  applied 
to  real  air  defense  intelligence  or  operations  problems,  they 
should  be  tested  to  determine  if,  where,  and  how  they  enhance 
human  performance.  This  Phase  I  Small  Business  Innovation 
Research  (SBIR)  project  may  well  have  succeeded  in  advancing  some 


14 


proof-of-novel-concept-demonstrations,  but  serious  questions 
remain  about  n^w  the  demonstration  prototypes  might  evolve  into 
working  systems. 
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2.0  THE  TARGET  DOMAIN;  AIR  DEFENSE 


2 . 1  The  Air  Defense  Intelligence  and  Operations  Continuum 

It  is  important  at  the  outset  to  recognize  the  integrated  nature 
of  our  strategic  intelligence,  warning  and  operations  processes. 
According  to  Latham  (1988),  "it's  a  mistake  to  think  of  air 
defense,  space  defense,  and  ballistic  missile  defense  as 
separate  missions  .  .  .  they  fit  together  naturally  as  parts  of  a 
single  package."  Similarly,  it  is  important  to  recognize  the 
integrated  nature  of  the  intelligence  and  operations  processes. 
Intelligence  is  an  input  to  operations  and  operations  feed  back 
to  intelligence.  While  there  are  often  bureaucratic  obstacles  to 
their  integration,  they  are  conceptually  and  functionally 
integrated. 

Our  research  assumes  both  realities.  We  assume  that  air  defense 
is  a  component  of  a  larger  strategic  mission  and  that  air  defense 
intelligence  and  operations  are  inextricably  linked. 


2 . 2  Air  Defense  Intelligence 


General  Robert  T.  Herres  has  suggested  that  we 


"  .  .  .  must  be  capable  of  providing  timely,  reliable  and 
unambiguous  warning  and  high  confidence  assessments  for 
posturing  U.S.  and  Canadian  forces  for  survivability  and  for 
force  execution.  This  requires  an  integrated  attack  warning 
and  assessment  system  with  the  capability  to  detect  and 
assimilate  the  overt  and  covert  indicators  of  a  coordinated 
Soviet  attack  —  even  when  the  attack  is  orchestrated  to 
create  confusion.  Put  another  way,  if  an  all-out  attack 
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were  launched  against  North  America,  it  would  most  likely  be 
an  integrated  attack.  The  Soviets  would  not  rely  solely  on 
ballistic  missiles,  but  would  probably  use  offensive  air 
forces,  cruise  missiles,  anti-satellite  weapons  and  other 
resources  at  their  disposal.  [NORAD]  must  accurately 
detect,  warn,  assess,  engage  and  respond  to  such  an 
integrated  attack." 


These  insights  suggest  clearly  the  essence  of  strategic  intel¬ 
ligence  and  warning;  the  air  defense  intelligence  process  can  be 
located  in  Herres's  challenge,  though  it  by  no  means  should  be 
seen  as  independent  of  the  other  dimensions  of  strategic 
intelligence . 


2.2.1  Generic  Intelligence  Collection,  Analysis  and 
Production  -  Intelligence  is  the  "poor  relation"  of  command, 
control,  and  communications.  Yet  without  intelligence  commanders 
cannot  generate  or  evaluate  options.  There  is  a  gap  between  the 
intelligence  and  operations  communities;  there  are  alternative 
intelligence  missions  across  the  national  intelligence  agencies 
and  the  Services, 

Figure  2.1  suggests  what  the  intelligence  process  assumes  (DIA, 
1984).  Collection,  production  and  dissemination  constitute  the 
primary  phases  of  the  process,  while  eight  steps  support  the 
three  phases. 

Figure  2.2  presents  the  forms  and  components  of  strategic 
intelligence  (Andriole,  1984).  Note  the  variety  of  potential 
intelligence  products  and  the  range  of  the  intelligence  mission. 
Analysts  are  responsible  for  a  variety  of  intelligence  products 
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Direction  and  Management 
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Figure  2. 1 :  The  Strategic  Intelligence  Process 


PAST  PRESENT  FUTURE 
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Figure  2.2:  The  Forms  and  Components  of  Strategic  Intelligence 


of  various  natures. 


Clarkson  (1981)  describes  the  strategic  intelligence  process  in  a 
series  of  stages  (see  Figure  2.3)  and  steps,  as  suggested  in 
Tables  2.1,  2.2,  and  2.3.  Figure  2.3  presents  the  "major 
functional  stages  in  strategic  analysis."  Tables  2.1,  2.2,  and 
2.3  describe  the  steps  that  comprise  the  analysis  process.  These 
steps  suggest  the  analytical  and  cognitive  processes  implemented 
during  the  intelligence  process.  They  also  suggest  how  the  air 
defense  intelligence  process  is  implemented. 

The  repeated  reference  to  indicators  and  correlations  is 
significant.  Intelligence  essentially  reduces  to  pattern 
recognition,  where  diverse  data  is  filtered  into  hypotheses  about 
likely  situations  and  behavior.  Put  another  way,  intelligence 
analysis  is  often  deductive,  where  evidence  is  sought  to  confirm 
or  disconfirm  hypotheses. 

Deductive  inference  frequently  dominates  the  intelligence 
process.  Analysts  look  to  data,  indicators,  and  groups  of 
indicators  and  data  —  activities  —  to  infer  likely 
situations  and  event  likelihoods.  Environments  are 
monitored  to  determine  (a)  what  is  happening,  (b)  how 
serious  it  is  and  (c)  when  the  critical  event  will  occur. 

Figure  2.4  from  Barclay,  et  al.  (1977)  presents  a  generic 
"hierarchical  inference  structure"  that  represents  the 
relationship  among  data,  indicators  and  activities,  while  Figure 
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Figure  2.3:  Clarkson’s  Functional  Steps  In  Strategic  Analysis 


step  1:  Review  Incoming  data 
Step  2:  Sort  Incoming  data 

Procedure  1 :  Geographic 

Procedure  2:  Military 


Table  2. 1 :  Strategic  Monitoring  Steps 


step  1 :  Match  Indicators  from  Monitoring  stage  against 
predetermined  threat  models 

Procedure  1 :  Call  up  preestablished  threat  models 
Procedure  2:  Match  Indicators  against  models 
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Table  2.2:  Strategic  Threat  Recognition  Steps 


St«9  I :  Compare  candidate  threats  in  unimpeded,  predictive  domain 

Pr«ca4«ra  I :  Translate  individual  threats  Into  predictive  format  based  on  information  categories 

a.  Wh*  (or  what) 

b.  (Could  do)  what 

c.  (To)  wham  (or  what) 
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Figure  2.4:  Generic  Hierarchical  Inference  Structure 


2.5  (also  from  Barclay,  et  al.,  1977)  illustrates  how  the 
hierarchical  inference  structuring  technique  can  be  used  to 
model,  monitor  and  project  North  Korean  behavior  toward  South 
Korea.  The  military  posture  model  is  comprised  of  a  variety  of 
indicators  and  activities  that  when  combined  in  alternative  ways 
will  determine  the  likelihood  of  each  of  the  four  hypotheses  that 
sit  on  top  of  the  hierarchical  inference  model. 

A  simpler  inference  model  is  the  influence  diagram  that 
recognizes  the  interrelationships  among  events  and  conditions. 
Influence  diagrams  are  conceptually  "causal,"  though  no  attempt  is 
made  to  verify  the  causality  statistically  or  mathematically. 
Rather,  the  focus  is  placed  upon  how  events  "chain-react"  to 
increase  (or  decrease)  the  likelihood  of  a  specific  event  or 
condition.  Figure  2.6  illustrates  the  influence  diagramming 
process,  while  Figure  2.7  presents  an  influence  diagram  of  the 
likelihood  of  having  to  evacuate  American  nationals  from  Lebanon 
(Andriole,  1984). 

Finally,  probability  trees  can  be  used  to  describe  the 
intelligence  process.  Figure  2.8  presents  a  model  of  the 
likelihood  of  a  Soviet  invasion  of  China  given  the  death  of  the 
Chinese  Premier. 

All  of  these  models  suggest  that  the  intelligence  process  is 
procedural  and  directed.  They  also  suggest  how  we  might  describe 
and  understand  the  air  defense  intelligence  process. 
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H-1  -  NO  MILITARY  ACTION 
H-2  -  SMALL  RAID;  ISLAND  ATTACK 
H-3  -  LIMITED  MILITARY  ATTACK 
H-4  -  ALL-OUT  WAR 


Figure  2.5:  Korean  Hierarchical  Inference  Structure 


Event  \ _ ^  Event  \  J  Event 


Figure  2.6;  Influence  Diagramming  Process 
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Figure  2.7:  Lebanon  Evacuation  Likelihood  Influence  Diagram 


Russia  Attacks 


I 
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Figure  2.8:  Soviet  Invasion  Like! Ihood/Probabtllty  Tree 


2.2.2  Air  Defense  Intelligence  Analysis  --  The  larger 


strategic  intelligence  process  —  comprised  of  space,  missile  and 
air  defense  —  assumes  current  and  future  collection,  analysis 
and  production  technology.  Covault  (1985)  presents  the  attack 
warning/attack  assessment  process  in  Figure  2.9,  a  process  that 
features  technology-based  collection  via  sensors  and  intelli¬ 
gence  . 

Air  defense  intelligence  focuses  on  the  atmospheric  threat,  that 
is,  the  manned  bomber  and  air-breathing,  non-ballistic  missile 
threat.  As  suggested  above,  however,  while  the  air  defense 
intelligence  mission  may  well  be  segmented  it  cannot  realisti¬ 
cally  be  restricted  to  the  atmospheric  threat.  Given  the 
likelihood  of  an  integrated  attack  it  must  be  presumed  by  all 
strategic  intelligence  analysts  that  if  one  set  of  indicators 
begin  to  move,  it  is  highly  likely  that  the  others  are  moving 
too. 

Air  defense  intelligence  is  supported  of  course  by  powerful 
warning  capabilities.  The  warning  function,  which  assumes  action 
and  short  lead-time,  is  the  intelligence  fail-safe.  Ideally, 
however,  assessments  can  be  made  long  before  an  over-the-horizon 
radar  detects  a  Blackjack  bomber.  Intelligence  is  intended  to 
extend  lead-time  via  analyses  of  phenomena  likely  to  occur  before 
the  atmospheric  threat  becomes  acute. 

The  need  for  accurate,  timely  and  reliable  air  defense 
intelligence  is  critical  since  it  is  extremely  difficult  to 


Figure  2.9:  Covault’s  Attack  Warning/ Attack  Assessment  Process 


detect  and  track  manned  bombers  and  cruise  missiles  "because  they 
are  not  limited  to  ballistic  trajectories"  (Gumble,  1986).  Over- 
the-horizon  backscatter  (OTH-B)  radar  capable  of  high  frequency 
operation  is  one  solution  to  the  detection  and  tracking  problem, 
though  it  is  unlikely  that  the  atmospheric  threat  can  be 
eliminated  in  the  near  or  even  distant  future. 

Intelligence  is  necessary  to  mitigate  the  warning  problem.  The 
desire  for  long  lead  time  presumes  a  multiple  indicator/activity 
approach.  Deductive  inferential  processes  are  implemented  by  the 
air  defense  intelligence  analyst  on  a  variety  of  interconnected 
levels.  Economic,  political  and  (especially)  military  indicators 
are  observed  and  correlated.  Unfortunately,  there  are  precious 
few  historical  precedents  available  to  the  air  defense  analyst, 
precedents  that  could  be  converted  into  inference-making 
"templates."  The  United  States  does  not  possess  perfect 
intelligence  about  Soviet  manned  bombers  and  cruise  missiles. 
Given  that  exercise  data  is  the  best  we  can  do,  our  inferences 
about  indicators  of  atmospheric  activity  are  twice-removed  from 
those  anchored  in  deep  historical  experience. 

Models  of  expected  precipitous  behavior  are  used  to  filter, 
interpret  and  correlate  air  defense  indicators;  missile  warning 
indicators  are  used  to  validate  air  defense  inferences,  and  vice 
versa . 


33 


2.2.3  Air  Defense  Intelligence  Requirements  -  From  a 
systems  design  perspective,  there  are  a  number  of  requirements 
that  together  constitute  a  requisite  set  of  functions  and  tasks 
that  must  be  performed  (either  manually  or,  ideally,  with 
computer-based  support).  In  terms  of  the  forms  and  components 
of  strategic  intelligence.  Figure  2.10  highlights  the  most 
pertinent  cells  in  the  matrix. 

Clarkson's  functional  outlines  of  the  stages  of  strategic 
intelligence  provide  a  template  for  the  identification  of  air 
defense  intelligence  requirements,  especially  as  they  pertain  to 
the  design  of  a  system  intended  to  support  intelligence  analysts. 
Tables  2.4,  2.5,  and  2.6  present  a  set  of  requirements  necessary 
to  develop  an  interactive  system  concept  (see  Section  5.0  for  the 
conversion  of  these  requirements  into  demonstration  prototypes 
with  reference  to  findings  from  cognitive  science  and  advanced 
information  technology) . 

Tables  2.4,  2.5,  and  2.6  do  not,  however,  explicitly  recognize 
the  interrelationships  among  indicators  across  strategic 
intelligence  functions.  Air  defense  intelligence  analysts  must 
correlate  indicators  with  missile  defense  indicators;  intelli¬ 
gence  models  must  be  integrated  if  they  are  to  be  effective  in 
diagnosing  deception. 

The  reason  why  the  integration  and  synthesis  of  strategic  models 
is  so  important  is  because  of  multiple  potential  origins  of 
threat.  A  bona  fide  strategic  attack,  as  suggested  by  Latham  and 
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Figure  2.10;  The  Forms  and  Components  of  Strategic  Air  Defense  Intelligence 
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Table  2.5;  Strategic  Threat  Recognition  and  System  Requirements 


Herres  above,  will  most  likely  be  ground,  air,  sea  and  space 
launched.  Air  defense  intelligence  analysis  cannot  be  conducted 
independent  of  ground,  sea  and  space-based  behavior.  Any  system 
designed  to  support  air  defense  intelligence  analysis  must  by 
function  also  support  ground,  sea  and  space  intelligence  analysis 
—  or  at  least  have  access  to  such  data,  models,  and  analytical 
systems  (see  Section  5.0). 

Air  defense  intelligence  must  also  satisfy  one  additional 
requirement:  easy  transition  to  warning  and  operations.  The 

intelligence/warning/operations  process  does  not  always  proceed 
smoothly.  Disconnects  often  exist  between  the  form  and  content 
of  intelligence  estimates  and  inferences,  disconnects  that 
prevent  operations  personnel  from  absorbing  the  essence  of  the 
analysis . 

There  are  "culture"  problems,  organization  problems,  and  even 
doctrinal  problems  that  constrain  the  communication  bandwidth 
between  intelligence  and  operations  personnel  (see  Section  2.4 
for  more  detail)  An  effective  support  system  must  widen  the 
communications  bandwidth  via  an  understanding  of  intelligence, 
warning,  and  operations  requirements. 

2.3  Air  Defense  Operations 

The  transition  from  intelligence  to  warning  to  operations  is 
often  far  from  smooth.  Yet  operations  are  dependent  upon 
intelligence  and  warning  triggers,  just  as  intelligence 
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should  be  produced  with  explicit  reference  to  operational 
realities.  This  section  examines  the  air  defense  operations 
process  and  identifies  a  set  of  operational  requirements 
necessary  for  the  design  and  development  of  computer-based 
systems  intended  to  support  intelligence,  warning  and  operations 
personnel . 

2.3.1  Generic  Operational  Dec i s i on-Mak i ng  -  There  are  any 
number  of  decision-making  models  and  paradigms.  These  models  and 
paradigms  share  several  things  in  common.  They  generally  offer  a 
sequence  of  stages  or  steps  that  decision-makers  normatively 
execute.  Built  into  the  models  are  assumptions  about  how 
decisions  can  and  should  be  made.  Note,  for  example,  the 
following  two  decision  process  models  in  Figure  2.11  from  Easton 
(1979).  The  model  on  the  left  is  one  that  assumes  decision¬ 
makers  will  "optimize,"  while  the  other  assumes  "satisficing" 

(see  Section  3.0  for  additional  insight  into  decision  process 
models  and  satisficing). 

Wohl’s  (1981)  Stimulus-Hypothesis-Option-Response  (SHOR)  model 
represents  yet  another  perspective  (see  Figures  2.12  and  2.13). 
Wohl's  model  expands  the  generic  decision  process  model  to 
include  insight  into  specific  decision-making  functions  and  the 
kind  of  information  necessary  to  implement  a  decision's  generic 
elements.  His  domain  is  tactical  Air  Force  planning  and 
decision-making,  but  the  model  is  applicable  to  a  wide  variety  of 
decision  modeling  problems. 


OPTIMIZING 


SATISFICING 


8 


Figure  2. 1 1 ;  "Optimizing"  and  "Satisficing"  Decision  Process  Models 
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GENERIC  ELEMENTS  FUNCTIONS  REQUIRED  INFORMATION  PROCESSED 
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Figure  2. 1 2:  Wohl  s  SHOR  Model 
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A  variety  of  other  analysts  have  approached  decision-making  from 
another  perspective.  Instead  of  modeling  the  process,  some  have 
modeled  the  range  of  problems  that  operational  decision-makers 
face.  Hopple  (1986)  combines  Wohl's  model  with  decision  problem 
characteristics  to  develop  the  matrix  in  Figure  2.14,  while 
Andriole  (1989a)  has  developed  a  three  dimensional  matrix  that 
defines  decision  problems  according  to  the  nature  of  tasks, 
decision-makers  ("users"),  and  the  organizational-doctrinal 
setting  in  which  the  problems  are  addressed  (see  Figure  2.15). 
Hermann  (1969)  has  developed  a  "situational  cube"  that  defines 
decision  problems  according  to  their  location  along  decision 
time,  threat  and  awareness  continuua,  as  suggested  in  Figure 
2.16. 

Biddle  (1989)  has  developed  a  list  of  decision/task  attributes 
that  help  identify  the  range  of  decision  problems  faced  by 
operational  personnel.  Table  2.7  presents  this  list  and  suggests 
how  decision-making  processes  will  vary  according  to  the  nature 
of  the  task  to  be  completed. 

Various  methods  and  approaches  have  been  advanced  to  describe  and 
prescribe  how  decision  alternatives  can  be  compared.  While  we 
are  all  susceptible  to  biases  and  other  cognitive  predispositions 
(see  Section  3.0),  we  also  search  for  ways  to  compare 
alternatives  in  order  to  select  the  "best"  one.  Decision 
analysts  suggest  multi-attribute  utility  theory  as  one  approach 
(see  Figure  2.17);  there  are  others.  Humans  seek  ways  to 
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Figure  2. 1 5;  Andriole's  Decision  Problem  Matrix 
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Figure  2.16;  Hermann’s  Situational  Cube 


•  Level  of  Abstraction  of  the  Task 


-  Skills,  Rules,  Knowledge 

--  Does  the  task  require  explicit  plan  development  and  selection? 

--  Is  the  task  solvable  by  application  of  standard  operating  procetkires? 

--  Does  the  task  require  specifically  identified  procedures  or  can  it  be 
perfxmed  with  little  thought? 

-  Deduction,  Induction.  Abduction 

--  Does  the  solution  of  the  task  require  the  development  of  new  data  or 
information  (Induction)? 

--  Does  the  task  rely  on  already  developed  Information  for  Its  solution 
(deduction)? 

--  Is  It  necessary'  to  generate  explanations  for  a  series  of  observations? 

-  Perceptual,  Medlattonal,  Communication  and  Motor 

--  Sensory  Dominant  -  Perceptual 
--  Information  Processing  and  Decision  Making  Tasks 
--  Communication  Dominant 
--  Mechanical  (Skill  Like)  Processes 

•  Level  of  Structure  to  the  Task 

-  Level  of  Structure 

--  Highly  structured,  solution  processes  well  understood  and  commonly 
Implemented  in  an  automated  system 
--  Less  well  strxtured  than  above  but  solution  processes  still  well 
understood  and  have  been  successfully  automated  previously 
--  Relatively  unstructured  problem  but  solution  processes  still  tractable 
-  -  Unstructured  problem .  solution  processes  poorly  understood 

-  Bounds  of  the  Problem 

--  All  variables  are  known  and  understood 
--  Unexpected  variables  seldom  Impact  solution  processes 
--  Unexpected  variables  often  Impact  results 

--  Currently  impossible  to  bound  the  problem  (Impossible  to  separate  from 
the  environment) 

•  Organizational  Aspects  of  the  Problem 

-  Safety/Responsiblllty/Traditlon/Polltics  require  that  a 

human  manage  the  process 

-  Safety/Responslbiltty/Tradltlon/PoHtlcs  are  not  overriding 

factors 
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•  Data  Aspects  of  the  Problem 

-  Amount  of  Data  Required 

-  -  Massive  amount  of  data  required;  beyond  capability  of  human  to 

absorb  or  manipulate 

--  Large  amount  of  data  required;  possibly  could  be  assimilated  by  dedicated 
human  operator 

--  Minimal  data  requirements;  easily  mastered  by  operator 

-  Accuracy  of  Available  Data 

--  Available  data  is  totally  accurate  or  accuracy  is  not  necessary  to  develop  a 
correct  solution 

--  Available  data  is  generally  accurate 

-  -  Available  data  is  suspect 

-  Precision  Required  of  the  Data 

-  -  High  precision  required  of  input  data  to  achieve  solution;  precision  beyond 

capacity  of  human  to  manipulate 
--  Average  precision  required  of  input  data 
--  Precision  of  data  not  a  factor  in  solution  to  problem 

-  Periodicity  of  Data 

--  Data  of'  ives  periodically  in  known  format,  and  in  known  ranges 
--  Data  arrives  within  general  constraints  of  time  and  format 

-  -  Data  arrives  randomly  and  unexpectedly 

-  Amount  of  Data  Available  vs.  Amount  of  Data  Required 

--  All  required  data  is  available  and  manipulatable 

--  All  critical  data  is  available  and  usable 

--  Decisions  must  be  made  without  knowledge  of  necessary  data 

-  Fuzziness  of  Data 

-  -  Data  sets  are  crisp  and  accepted  as  an  accurate  representation  of  reality 

-  -  Most  data  is  understood  and  stable 

--  Data  Is  not  well  understood  and/or  is  unreliable 

-  Specificity  of  Data 

--The  CBta  Is  naturally  or  easily  quantifiable 

-  -  The  data  is  difficult  to  quantify  without  losing  critical  infxmation  or 

making  difficult  assumptions 
--  The  data  cannot  be  legitimately  quantified 

-  Storage,  Recall,  and  Manipulation  of  Data  Required 

--  Massive  amounts  of  data  are  required  to  be  stored,  recalled  on  short 
notice,  and  manipulated 

--  Quantity  and  manipulation  of  data  are  not  a  factor  in  the  problem  solution 
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Time  Requirements  of  the  Problem 


-  Time  Requirements 

—  Real-time  problem 

-  -  Severe  time  constraints 

—  Time  constraints  do  not  Impact  on  problem 

-  Computational  Intensity 

--  Solution  to  problem  requires  unacceptable  amount  of  time,  effort,  or 
precision  to  be  done  manually 
--  Problem  difficult  but  tractable  to  solve  manually 
--  Little  computation  involved  in  solving  problem 


Table  2.7:  Biddle's  Task  Taxonomy 


Overall 

Evaluation  Criteria 
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Figure  2.17:  Multi-Attribute  Utility  Assessment  Model 


optimize,  "satisfice,"  and  survive  in  a  variety  of  operational 
environments;  the  generic  decision  models  presented  above  are 
just  that.  In  spite  of  their  analytical  origins,  they  are 
adaptive  to  decision  situations,  makers,  and  constraints.  They 
are  also  —  ultimately  —  only  compasses  to  domain  specific 
research . 

2.3.2  Air  Defense  Operations  -  Like  a  great  deal  of 
military  decision-making,  air  defense  operations  are  to  an  extent 
template-driven.  There  are  a  series  of  if-then  procedures  that 
determine  at  least  the  subset  of  decisions  and  plans  that  should 
be  implemented  given  a  set  of  environmental  conditions. 

Strategic  and  tactical  decision-making  is  frequently  template- 
driven  and  frequently  routinized  via  pre-conceived  decision 
options . 

Operational  environments  like  air  defense  intelligence  and 
operations  (as  well  as  missile  and  space  defense)  are  thought  to 
be  boundable.  A  finite  number  of  threats  is  assumed  to  drive  a 
finite  number  of  options.  The  overall  strategic  defense  of  the 
United  States  is  anchored  in  a  set  of  pre-determined  responses  to 
a  relatively  finite  range  of  adversary  actions  (and  reactions). 

Any  requirements  analysis  of  an  assumed  bounded  domain  must  first 
address  the  nature  of  the  boundedness  and  the  finite  assumptions 
that  define  it.  There  are  situations,  however,  that  will  fall 
outside  the  problem  boundaries,  situations  that  any  "system"  must 
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be  capable  of  addressing.  To  a  great  extent,  the  problem  is 
analogous  to  the  difficulties  inherent  in  the  movement  from 
deductive  to  inductive  reasoning.  Marty  of  our  strategic  and 
tactical  systems  are  quite  capable  of  responding  deductively  to 
sets  of  stimuli,  but  only  remotely  capable  of  handling 
unpredictable  events.  Our  analysis  of  strategic  air  defense 
assumes,  on  the  one  hand,  that  sets  of  responses  to  expected 
event  stimuli  exist  and,  on  the  other,  that  any  advanced  system 
concept  must  deal  directly  with  unexpected  event  stimuli. 

2.3.3  Air  Defense  Operational  Requirements  -  McFarren,  et 
al.  (1988)  have  modeled  the  air  defense  operations  process  in  a 
series  of  modified  flow/Petri  diagrams  that  represent  the 
functions,  tasks,  and  sub-tasks  that  together  constitute  air 
defense  operations.  Several  of  these  diagrams  appear  in  Figures 
2.18,  2.19,  2.20  and  2.21  (the  complete  set  of  operations 
diagrams  appear  in  Appendix  A) .  Such  diagrams  permit  the  dynamic 
analysis  of  the  air  defense  operations  process  and  ^-ogether 
represent  air  defense  operations  requirements.  We  have  built 
directly  upon  this  work  and  have  extended  the  task/process 
representation  via  findings  in  cognitive  science  and  the 
capabilities  of  advanced  information  technology  (see  Section 
5.0).  We  have  integrated  assumptions  about  the  bounded/ 
unboundedness  of  the  domain  and  placed  operational  requirements 
along  the  air  defense  intelligence/warning/decision-making 
continuum . 
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Figure  2.18;  McFarren’s  IDEF  ADOC  Model.  Example  1 
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Figure  2. 1 9:  McFarren’s  IDEF  ADOC  Model.  Example  2 


Figure  2.20:  McFarren’s  IDEF  ADOC  Model.  Example  3 
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2.4  The  Air  Defense  Intelligence/Operations  Interface 

As  suggested  throughout  this  report,  we  regard  the  air  defense 
process  as  a  self-contained  continuum  consisting  of  intelligence, 
warning  and  decision-making  *{ffT9Cresses  and  as  part  of  the  much 
larger  strategic  intelligence  and  operations  process.  We  have 
argued  that  it  is  impossible  to  think  about  air  defense 
operations  independent  of  intelligence  estimates  and  that  it  is 
impossible  to  think  about  air  defense  independent  of  missile 
warning,  SLBM  warning  and  space  defense.  This  perspective 
complicates  our  research  since  it  requires  the  multiple  level 
modeling  of  strategic  defense. 

The  system  concepts  described  in  Section  5.0  are  anchored  in 
these  assumptions.  The  intelligence/operations  interface  —  in 
air  defense  and  in  other  intelligence/operations  domains  (like 
missile  defense)  —  is  conceptually  and  functionally  linked.  Our 
demonstration  prototypes  suggest  how  information  technology  — 
tempered  with  findings  from  cognitive  science  —  can  be  leveraged 
to  widen  the  communications  bandwidth  among  all  of  the  analysts 
along  the  continuua. 

Requirements  for  the  design  of  systems  intended  to  address  the 
interface  must  be  derived  first  from  intelligence  and  operations 
requirements  and  then  from  models  of  how  the  two  processes  are 
linked.  The  McFarren,  et  al.  (1988)  modeling  technique  used  to 
model  the  Air  Defense  Operations  Center  (ADOC)  could  certainly 
be  applied  to  the  intelligence  side  of  the  process  and  the 
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intelligence/operations  interface.  An  IDEF  model  of  the  entire 
process  (at  all  strategic  defense  levels)  would  contribute 
significantly  to  our  understanding  of  the  individual  and  combined 
processes . 

We  focus  here  on  what  intelligence  needs  from  operations  and  on 
what  operations  needs  from  intelligence.  Section  3.0  discusses 
the  cognitive  needs  in  detail;  suffice  it  to  say  here  that 
optimal  operational  decision-making  requires  an  understanding  of 
the  intelligence  analysis  and  production  process,  and  that 
effective  intelligence  analysis  and  production  can  be  enhanced 
via  an  understanding  of  the  uses  made  of  intelligence  by 
operations  analysts. 

Functionally,  the  intelligence/operations  interface  can  be 
described  in  matrices,  as  suggested  in  Figure  2.22  modified  from 
Fitzgerald,  et  al.  (1988),  Belden  (1977)  has  developed  a 
"decision  stairway"  that  integrates  intelligence  and  operations 
into  the  likelihood  of  war  (Figure  2.23),  while  Brown,  et  al. 
(1975)  demonstrates  in  Figure  2.24  how  decision  analytic  models 
can  incorporate  intelligence  into  operational  decision-making 
processes.  But  all  of  these  notions  and  models  assume 
communication  between  intelligence  and  operations  analysts, 
communication  that  can  be  enhanced  via  the  marriage  between 
cognitive  science  and  information  technology. 


59 


60 


Figure  2.22:  Intel  I  Igence/Operat Ions  Interface 


EXECUTION 


Igure  2.23:  Belden  s  "Decision  Stairway" 
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Figure  2.24:  Integrated  Intel  11  gence/Opt ion  Selection  Process 


2.5  Air  Defense  System  Concept  Design 

Air  defense  intelligence  and  operations,  and  the  intelligence/ 
operations  interface,  constitute  the  domain  from  which  we  have 
extracted  a  set  of  descriptive  requirements.  But  we  have  also 
extended  the  requirements  into  the  realm  of  the  prescriptive.  It 
should  be  noted  that  conventional  systems  design  does  not 
generally  allow  for  the  alteration  of  established  modus  operandi 
or  doctrine.  However,  given  the  nature  of  the  research  at  hand 
and  the  premium  placed  lip’'e  upon  innovation,  we  have  implemented 
an  "unconventional"  design  strategy  and  developed  some  system 
concepts  that  do  not  explicitly  track  with  doctrine.  Nor  are  the 
proposed  user-computer  interfaces  (UCIs)  conventional,  since  they 
are  intended  to  demonstrate  what  is  possible,  not  what  exists  or 
might  incrementally  be  introduced. 
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3.0  APPLIED  COGNITIVE  ENGINEEERING 


3 . 1  Cognitive  Engineering  in  Perspective 

Cognitive  engineering  is  a  relatively  new  field  of  inquiry  that 
calls  for  the  design  and  development  of  computer-based 
information  systems  consistent  with  what  we  know  about  how  humans 
process  information  and  make  decisions.  According  to  Norman 
(1986),  cognitive  engineering  is  .  a  type  of  applied  cognitive 
science,  trying  to  apply  what  is  known  from  science  to  the  design 
and  construction  of  machines."  It  represents  the  attempt  to  base 
system  design  on  cognitive  research  findings  regarding  how  people 
think;  to  aid  —  not  replace  —  the  human  problem-solver;  to 
design  systems  that  utilize  human  strengths  and  compensate  for 
our  cognitive  limitations;  to  more  systematically  integrate 
display  formats  and  analytical  methods  with  intuitive  thought. 

Put  quite  simply,  the  goal  of  cognitive  engineering  is  to  have 
human  cognitive  processes,  not  computer  technology,  drive  the 
human-computer  interaction  and  problem-& ->lvJ ng  processes. 

This  section  of  the  report  overviews  the  findings  from  a  number 
of  different  areas  of  cognitive  research  that  are  potentially 
applicable  to  the  cognitive  engineering  of  advanced  information 
syscetus.  In  contrast  to  many  important  cognitive  reseach 
efforts,  we  do  not  focus  on  the  cognitive  engineering  of  the  user 
interface  for  the  sole  purpose  of  better  using  a  software 
package,  such  as  a  editor  (Card,  ^  Newell,  1983)  or  a 
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word  processor  (Norman,  1986)  or  a  spreadsheet  (Olson  and  Nilson, 
1987).  Instead,  our  focus  is  on  enhancing  organizational 
problem-solving?  that  is,  designing  the  user  interface  to 
advanced  information  systems  so  that  they  improve  human  ability 
to  perform  the  complex  inference  and  decision-making  tasks  found 
in  organizational  settings  (Adelman,  1990).  Our  particular 
organizational  focus  is  strategic  command  and  control  and, 
specifically,  strategic  air  defense. 

For  expository  purposes,  we  categorize  strategic  air  defense 
problems  and  tasks  by  the  two  principal  organizational  units: 
intelligence  and  operations.  The  goal  of  intelligence  is 
effective  situation  assessment;  that  is,  monitoring  the  strategic 
environment  and  warning  friendly  forces  of  changes  in  sufficient 
time  for  effective  action.  The  goal  of  operations  is  the 
development  and  execution  of  effective  actions  for  the  situation 
at  hand.  The  kinds  of  intelligence  tasks  toward  which  our  review 
of  the  cognitive  literature  is  directed  include  discriminating 
between  "signals"  and  "noise"  in  the  target  environment ( s ) ; 
assessment  of  events  and  conditions;  inferences  about  possible 
enemy  intentions  and  courses  of  action;  and  identification  of 
high  probability  crisis  situations.  Operations  tasks  include, 
for  example,  assessment  of  friendly  forces?  option  generation; 
evaluation  of  alternative  options;  and  the  selection  and 
implementation  of  a  given  option. 

Consistent  with  the  above  discussion,  the  cognitive  research 
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areas  are  categorized  under  the  headings  of  intelligence  and 
operations.  In  addition,  since  effective  communication  within 
and  between  intelligence  and  operations  is  essential  to  high 
levels  of  organizational  performance,  the  third  (and  last) 
section  of  our  review  will  consider  the  cognitive  research  on 
understanding  and  communicating  problem-solving  processes. 

It  is  important  to  note  that  for  each  cognitive  research  area  we 
will  briefly  overview  the  types  of  tasks  used  in  performing  the 
research.  Although  we  realize  that  the  reader's  primary  concern 
is  learning  about  the  findings  of  different  areas  of  research, 
there  is  substantial  empirical  research  (e.g.,  Hammond  et  al., 
1975;  Hogarth,  1981;  Payne,  1982)  demonstrating  that  human 
information  processing  and  decision-making,  as  found  in 
intelligence  and  operations,  are  extremely  sensitive  to  task 
characteristics.  Carroll  (1989)  has  even  gone  so  far  as  to 
question  the  viability  of  a  cognitive  engineering  discipline 
because  "the  science  base  in  which  design  deductions  must  be 
anchored  is  too  general  and  too  shallow  vis-a-vis  specific 
contexts  and  situations  in  the  world." 

Although  we  disagree  with  Carroll,  we  caution  readers  against 
blindly  generalizing  basic  research  findings  to  complex 
organizational  settings.  The  focus  of  the  larger  project,  for 
which  this  review  is  but  a  part,  is  directed  toward  (a) 
identifying  display  technology  that,  on  the  basis  of  the  review 
presented  herein,  should  enhance  cognitive  processing,  and  (b) 
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developing  storyboard  representations  demonstrating  the  use  of 
cognitively-selected  display  technology.  Further  research  would 
need  to  develop  actual  prototype  systems  and  perform  empirical 
evaluations  to  determine  whether  performance  is  indeed  improved, 
as  hypothesized,  in  a  strategic  air  defense  setting.  We  focus  on 
the  different  tasks  used  in  basic  research  as  a  means  of  more 
effectively  moving  tcv.’ard  that  end. 

3 . 2  Intelligence ;  Situation  Assessment 

This  section  overviews  the  following  areas  of  cognitive  research, 
all  of  which  are  applicable  to  the  inference  processes  inherent 
in  situation  assessment: 

•  Social  judgment  theory; 

•  Cognitive  heuristics  and  biases;  and 

•  Analogical  reasoning. 

3.2.1  Social  Judgment  Theory  ( SJT)  -  SJT  was  developed  by 
Kenneth  R.  Hammond  and  his  colleagues  and  students,  and  extends 
the  theory  of  probabilistic  functionalism  developed  by  Egon 
Brunswik,  a  perception  psychologist,  to  the  realm  of  judgment  and 
decision  behavior.  For  an  overview,  the  interested  reader  is 
referred  to  the  paper  by  Hammond  et  al.  (1975)  and  the  edited 
volume  in  Hammond's  honor  by  Brehmer  and  Joyce  (1989).  For  its 
potential  application  to  intelligence  analysis,  the  reader  is 
referred  to  Adelman  et  al.  (1981)  and  Phelps  (1983). 
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Figure  3.1  represents  the  extension  of  Brunswik ' s  lens  model  to 
the  case  of  intuitive  prediction.  The  goal  of  human  judgment 
(Ys)  is  to  accurately  predict  environmental  criteria  (Ye).  These 
criteria  (Ye)  may  be  the  causes  of  observed  effects  or  future 
states  of  the  world.  In  the  domain  of  strategic  intelligence, 
this  may  be  the  enemy's  actual  intent  or  action?  for  example, 
whether  the  enemy  actually  intends  to  or  already  has  launched  an 
attack.  Ys  represents  the  judgmental  counterpart  to  Ye.  In 
strategic  intelligence,  for  example,  it  may  represent  the 
analyst's  (or  commander's)  judgment  as  to  whether  or  not  the 
enemy  actually  intends  to  or  has  already  launched  an  attack. 

People  use  information  or  cues  (Xi)  to  make  these  judgments.  The 

cues  may  be  acquired  from  the  immediate  environment  and  our 

memory  of  past  events.  The  cues  are  processed  (i.e.,  utilized) 

by  the  person  according  to  decision  heuristics  to  produce  the 

judgment  (Ys).  As  will  be  discussed  throughout  this  review, 

people  utilize  a  large  repertoire  of  decision  heuristics  to 

combine  information  into  a  judgment.  Moreover,  in  an  effort  to 

be  accurate,  people  try  to  utilize  the  cue  information  (r  )  in 

s,  i 

a  manner  that  matches  the  information’s  environmental  validity 

(r  ).  That,  is,  people  try  to  rely  on  valid  predictors  of 
e,  i 

environmental  criteria  (Ye).  In  strategic  intelligence,  this  is 
equivalent  to  saying  that  analysts  attempt  to  utilize  the  best 
predictors  of  enemy  intent.  To  the  extent  that  they  are  able  to 
do  so,  the  more  accurate  their  judgments  (Ys)  of  the 
environmental  event  (Ye)  will  be. 
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Achievement 


lei  Representing  the  Task  and  Person 


SJT  research  has  identified  three  findings  of  particular 
importance  to  the  cognitive  engineering  of  advanced  information 
systems  for  supporting  strategic  intelligence.  First,  specific 
task  characteristics  significantly  affect  people ' s  ability  to 
perform  complex  inference  tasks .  Second ,  people  have  poor  insight 
into  their  inferential  process .  Third,  information  systems 
providing  cognitive  feedback  can  significantly  improve  task 
performance .  Before  considering  these  findings  in  more  detail, 
we  first  turn  to  discuss  (a)  the  characteristics  of  inferential 
tasks,  particularly  as  studied  within  the  SJT  paradigm,  and  (b) 
the  lens  model  equation,  which  has  been  developed  to  quantify  the 
relation  between  the  environmental  and  cognitive  systems 
represented  in  Figure  3.1. 

3. 2. 1.1  Task  Characteristics  -  SJT  research  has 

focused  on  why  inferential  tasks,  like  strategic  intelligence, 

are  difficult  even  for  experts  in  their  (appropriate)  domain.  In 

particular,  we  focus  on  five  task  characteristics  here.  First, 

by  themselves,  individual  cues  are  seldom  perfectly  valid 

predictors  of  Ye.  Rather,  the  relationship  between  individual 

cues  and  the  environmental  criterion  is  typically  probabilistic 

(i.e.,  r  §  1).  Consequently,  any  given  piece  of  information 
s,  i 

is  not,  by  itself,  perfectly  diagnostic  of  the  event  that  caused 
it  (or  that  which  is  to  be  predicted).  This  is  compounded  by  the 
fact  that,  second,  information  is  not  always  reliable.  That  is 
why  there  is  always  such  a  concern  in  intelligence  analysis 
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regarding  the  reliability  of  the  source  of  the  presented 
information.  It  is  important  to  note  that  reliability  and 
validity  are  two  different  concepts.  A  piece  of  information  can 
be  100%  reliable  and  eitiier  totally  diagnostic  (100%  validity)  or 
undiagnostic  (0%  validity)  in  predicting  Ye.  However,  the  less 
reliable  the  information,  the  less  valid  it  is  because  of  the 
inherent  uncertainty  (i.e.,  error)  in  the  information  itself. 

In  addition  to  the  validity  and  reliaoility  of  the  cues,  the 
functional  relations  between  individual  cues  and  the  criterion  Ye 
may  assume  a  variety  of  forms.  For  example,  there  may  be  a 
positive  or  negative  linear  relationship  between  values  on  a  cue 
and  the  criterion.  Or,  in  contrast,  there  may  be  nonlinear 
relationships,  such  as  U-shaped,  inverted-U  shaped,  or  S- 
functions.  Fourth,  there  are  various  heuristics  for  combining 
the  multiple  pieces  of  information  (cues)  into  a  judgment  (e.g., 
additively  or  according  to  specific  patterns).  Moreover,  it  is 
seldom  clear  in  complex  inference  tasks  which  combination  rule  is 
the  best  one  for  predicting  Ye.  Finally,  overall  task 
predictability  when  using  all  the  cues  is  seldom  perfect. 

Patterns  of  data  (Xi)  that  may  have  correctly  predicted  Ye  in  the 
past  may  fail  to  do  so  in  the  future.  Human  judgment  is  thus  not 
only  difficult  because  of  limited  human  information  processing 
capabilities,  but  also  because  of  the  very  nature  of  inferential 
tasks  themselves. 
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3. 2. 1.2  Lens  Model  Equation  -  The  Lens  Model  Equation 
(LME)  presented  below  quantifies  the  re3 at ionships  between  the 
environmental,  and  cognitive  systems  represented  in  Figure  3.1. 
Since  linear  models,  such  as  multiple  regression  and  analysis  of 
variance,  have  been  routinely  used  in  SJT  research,  they  will  be 
assumed  in  the  description  of  the  LME  parameters  although,  as 
will  become  clear,  the  parameters  represent  more  general 
concepts : 

r  =  GR  R 
a  s  e 

The  typical  judgment  task  consists  of  presenting  a  person  with  a 

number  of  cases  representing  the  judgment  problem.  Each  case  (or 

profile)  for  consideration  consists  of  a  mix  of  values  on  the 

several  cues  (Xi)  being  used  to  make  a  judgment  (Ys)  of  the 

environmental  criterion  (Ye).  r  is  the  correlation  between  the 

a 

person's  judgments  (Ys)  and  the  environmental  criterion  (Ye)  over 
a  number  of  cases.  Statistically,  it  represents  the  person’s 
accuracy  in  predicting  the  criterion  Ye. 

In  most  SJT  research,  G  represents  the  correlations  between  the 
"best  fitting"  (using  a  "least  squares"  procedure)  linear  model 
predicting  the  person's  judgments  and  the  "best  fitting"  linear 
model  predicting  the  environment  criterion.  Conceptuaj.ly,  G 
represents  the  level  of  correspondence  (or  match)  between  the 
task  and  the  person,  and  is  considered  a  measure  of  knowledge. 

To  the  extent  that  a  person  can  accurately  assess  the 
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diagnosticity  of  individual  cues,  their  functional  relations  with 

Ye,  and  the  heuristics  for  combining  cue  information,  the  greater 

their  knowledge  (G)  of  the  task  and  the  better  their  performance 

(r  ). 
a 

R  is  the  correlation  between  the  person's  judgments  and  the 
s 

predicted  values  of  the  judgments  based  on  the  (linear)  model  of 

the  person.  It  is  typically  considered  as  the  overall 

consistency  or  "cognitive  control"  exhibited  by  the  person  in 

making  his/her  judgment  because  it  is  analogous  to  assessing 

whether  a  person  will  give  the  same  judgment  for  the  same 

information  seen  a  second  time.  Finally,  R  is  the  correlation 

e 

between  the  criterion  and  the  predicted  values  of  the  criterion 
based  on  the  (linear)  environmental  model.  Consequently,  it 
represents  the  overall  predictability  of  the  environmental 
system,  and  the  extent  to  which  a  cause  will  always  generate  the 
same  observable  cues. 

The  parameters  of  the  LME  provide  important  insights  into  the 

nature  of  many  types  of  judgment  tasks  and  people's  ability  to 

learn  and  perform  them  well  both  in  and  outside  of  the 

laboratory.  First,  G  (i.e.,  knowledge)  and  R  (i.e.,  cognitive 

s 

control)  are  statistically  independent.  Consequently,  it  is 
possible  to  perform  quite  poorly  even  if  one  knows  what  to  do  if 
one  cannot  do  it  consistently.  Second,  sets  an  upper  bound 
on  achievement  even  if  knowledge  and  cognitive  control  are 
perfect.  In  other  words,  task  predictability  is  as  important  a 
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determinant  of  judgment  performance  as  our  information  processing 
capability  and/or  the  support  systems  that  we  develop  to  improve 
it . 


3. 2. 1.3  Important  Findings  -  As  suggested  above,  SJT 

research  has  identified  three  findings  of  particular  importance 

to  the  cognitive  engineering  of  advanced  information  systems  for 

supporting  strategic  intelligence.  First,  specific  task 

characteristics  significantly  affect  people ' s  ability  to  perform 

complex  inference  tasks  ( r  Moreover,  the  LME  permits  one  to 

a 

assess  whether  the  reason  is  poor  knowledge  (G)  and/or  cognitive 

control  (R  ).  For  example,  Hammond  (1971)  demonstrated  that  it 
s 

was  extremely  difficult  for  people  to  learn  nonlinear  relations 

(e.g.,  inverted  U-shaped  functions)  versus  linear  ones. 

Moreover,  Hammond  and  Summers  (1972)  showed  that  even  when  they 

learned  the  nonlinear  relations  as  well  as  the  linear  ones  (i.e., 

G  was  the  same) ,  achievement  (r  )  was  still  lower  for  the 

a 

nonlinear  ones  because  of  significantly  lower  cognitive  control 

(R  )-  That  is,  it  was  just  harder  for  people  to  do  what  they 
s 

wanted  to  do  cognitively. 

Adelman  (1981)  and  numerous  others  have  shown  that  cognitive 

control  (R  )  and  achievement  (r  )  are  significantly  decreased  by 
s  a 

decreasing  task  predictability  (i.e.,  lowering  R  ).  Yet,  there 

e 

is  minimal  (if  any)  affect  on  knowledge  (G).  This  appears  to 
occur  because  people  act  as  if  inferential  tasks  are  perfectly 
predictable.  More  generally,  people  seem  to  rely  on  causal 
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schemas  (versus  probability  statistics)  to  explain  observed 

events  (i.e.,  Xi).  Generalizing  to  the  domain  of  strategic 

intelligence,  this  suggests  that  the  inferential  performance  of 

analysts  can  be  significantly  hampered  by  the  effective  use  of 

deception  that  adds  extensive  noise  to  the  system  and/or  is 

designed  to  create  incorrect  causal  patterns.  Advanced 

information  systems  need  to  be  engineered  to  help  users  ignore 

the  noise  (1  -  R  )  and  stay  focused  on  their  knowledge  of  the 

e 

relations  between  the  observed  data  (Xi)  and  the  (potentially) 
multiple  causes  (Ye)  of  it. 

Although  not  performed  explicitly  within  the  SJT  framework, 
Einhorn  and  Hogarth  (1986)  and  Hogarth  (1987)  have  further 
advanced  this  research  by  focusing  on  three  critical  aspects  of 
causal  reasoning.  First,  their  research  indicates  that  people 
make  judgments  of  probable  cause  on  the  basis  of  a  "causal  field 
that  is  basically  analogous  to  a  perceptual  field.  Judgmental, 
like  perceptual  processes  are  attuned  to  differences;  therefore, 
the  relevance  of  potential  causes  depends  on  whether  they  are 
considered  as  differences  in  the  problem  context.  Second,  peopl 
use  various  imperfect  indicators  of  causal  relations  called 
"cues-to-causality . "  These  cues  include,  for  example,  temporal 
order,  covariation,  contiguity  in  time  and  space,  and  similarity 
of  cause  and  effect;  and,  third,  the  confidence  people  place  in 
a  causal  explanation  is  affected  by  the  extent  to  which  they  can 
imagine  plausible  scenarios  for  both  it  and  alternative 
explanations.  As  Hogarth  (1987)  points  out  in  his  discussion  of 


creativity  in  problem  solving,  "...use  of  both  causal  fields  and 
the  cues-to-causality  help  the  mind  establish  order  out  of  the 
mass  of  information  with  which  it  is  confronted.  On  the  other 
hand,  this  order  is  bought  at  the  cost  of  being  able  to  perceive 
alternative  problem  formulations  (i.e.,  causal  fields)  and 
potential  causal  candidates." 

The  second  important  finding  of  SJT  research  is  that  people  have 
poor  insight  into  their  inferential  process .  For  example,  it  has 
been  consistently  shown  that  people  think  they  use  more 
information  than  they  actually  do,  or  that  is  required  to 
accurately  predict  their  judgments.  In  particular,  people  tend 
to  underestimate  the  weight  they  place  on  important  cues  and 
overestimate  the  weight  they  place  on  unimportant  cues  when 
compared  to  the  (highly  predictive)  weights  obtained  from  a 
multiple  regression  analysis  of  their  judgments  (e.g.,  see  Cook 
and  Stewart,  1975).  Moreover,  experts  appear  just  as  susceptible 
to  poor  insight  as  novices  (e.g.,  see  Slovic  et  al.,  1972).  Poor 
insight  will  be  considered  in  more  detail  in  the  Communication 
section,  which  is  the  last  section  of  this  review,  because  it  has 
substantial  implications  for  interpersonal  understanding  and 
group  decision  making.  For  a  detailed  review  that  addresses  this 
issue  beyond  just  SJT,  see  Nisbett  and  Wilson,  1977. 

Third,  information  systems  providing  cognitive  feedback  can 
significantly  improve  task  performance.  Cognitive  feedback  is 
oriented  to  providing  people  with  information  regarding  how  they 
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use  the  cues  to  reach  a  judgment-  (e.g.,  r  weights  and 

function  forms)  and  how  they  should  use  the  information  (e.g., 
the  task  weights  and  functions).  Hammond  et  al.  (1975)  have 
argued  that  one  of  the  unique  contributions  of  SJT  research  is 
the  empirical  demonstration  that  cognitive  feedback  results  in 
significantly  higher  levels  of  performance  than  outcome  feedback 
(i.e.,  receiving  the  correct  answer).  The  reason  is  three-fold. 
First,  as  discussed  above,  people  have  poor  insight  into  how  they 
actually  use  information  to  make  a  judgment.  Second,  they  have 
poor  insight  into  how  they  should  use  the  information.  Newton 
(1965)  showed  that  information  about  the  task  (i.e.,  the 
environmental  side  of  the  lens  model)  is  significantly  more 
beneficial  than  information  about  the  person  (i.e.,  the  cognitive 
side  of  the  lens)  for  performance  in  complex  inference  tasks;  and 
third,  as  discussed  above,  people  often  get  distracted  by  the 
noise  in  the  system.  Cognitive  feedback  keeps  them  focused  on 
important  cue-criterion  relations. 

SJT  research  has  extended  the  focus  on  cognitive  feedback  to 
group  decision  making.  This  extension  will  be  considered  in  the 
Communication  section,  which  is  the  last  section  of  this  review. 
What  is  important  to  note  here  is  that  research  has  demonstrated 
that  cognitive  feedback  of  human  inferential  processes  has 
significantly  improved  interpersonal  understanding  and  subsequent 
performance  over  more  traditional  approaches,  like  talking  about 
the  problem  without  cognitive  feedback.  Moreover,  this  has  been 
extensively  demonstrated  in  the  laboratory  (e.g.,  Hammond  and 
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Brehmer,  1973)  and  in  real  world  applications  (e,g.,  see  Haminond 
and  Adelman,  1976;  Hammond  and  Grassia,  1985;  Rohrbaugh,  1988). 

In  SJT  research,  the  quantitative  and  qualitative  characteris¬ 
tics  of  the  task,  that  is,  the  environmental  side  of  the  lens 
model,  is  used  as  a  standard  by  which  to  compare  human 
information  processing,  that  is,  the  cognitive  side  of  the  lens. 
High  levels  of  achievement  depend  on  (1)  knowledge  (G),  that  is, 
matching  our  use  of  the  information  to  how  we  should  use  it,  and 
(2)  cognitive  control,  that  is,  implementing  our  knowledge  in  a 
consistent  fashion.  However,  the  types  of  tasks  used  in  SJT 
represent  just  some  of  the  many  different  types  of  tasks  that 
have  been  used  by  cognitive  scientists  studying  human  inference. 
We  now  turn  to  consider  other  types  of  inferential  tasks  and, 
more  generally,  the  results  of  research  focusing  on  cognitive 
heuristics. 


3.2.2  Cognitive  Heuristics  -  Cognitive  heuristic  research 
has  used  probability  and  statistical  theory  as  a  normative 
standard  for  evaluating  human  inferential  performance. 
Probability  and  statistical  theory  represent  powerful  rules  for 
logically  combining  uncertain  information  into  an  inference. 
Moreover,  many  advanced  information  and  decision  systems  used 
these  rules.  Therefore,  from  a  cognitive  engineering 
perspective,  it  is  important  to  assess  whether  humans  try  to 
emulate  these  rules  or  use  other  types  of  processing  heuristics. 
It  is  important  to  emphasize  that  we  are  not  saying  that  people 
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should  use  the  rules  of  probability  and  statistical  theory  but, 
rather,  whether  they  do  so. 

An  overwhelming  finding  is  that  human  decision  behavior 
systematically  deviates  from  (or  is  "biased"  when  compared  to)  a 
normative  model  that  is  assumed  to  be  the  optimal  way  to  make  the 
decision  under  investigation.  According  to  Hogarth  (1987) 

"people  do  not  possess  intuitive  'calculators’  that  allow  them  to 
make  what  one  might  call  'optimal'  calculations.  Rather,  they  use 
fairly  simple  procedures,  rules  or  'tricks'  (sometimes  called 
’heuristics’)  in  order  to  reduce  mental  effort."  Judgmental 
heuristics,  and  the  biases  they  often  spawn,  are  the  results  of 
human  effort  to  understand  and  master  our  environment  given 
limited  information  acquisition,  retention,  and  processing 
capacities . 

Research  has  demonstrated  that  the  perception  of  information  is 
not  comprehensive  but  selective.  As  Hogarth  (1987)  points  out,  it 
has  been  estimated  that,  for  example,  only  about  l/70th  of  what 
is  present  in  the  visual  field  can  be  perceived  at  one  time.  How 
do  we  know  what  to  select?  The  answer  is  that  we  anticipate 
information  on  the  basis  of  our  causal  model  of  the  environment. 
Similarly,  memory  is  limited;  we  recall  but  a  small  part  of  the 
information  we  initially  acquire.  Moreover,  current  theories 
support  the  view  that  memory  does  not  access  information  in  its 
original  form  but,  rather,  works  by  a  process  of  associations 
that  reconstruct  past  events  and  fragments  of  information  that 
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are  typically  consistent  with  our  causal  model  of  the 
environment.  Finally,  research  indicates  that  considerable  mental 
activity  involves  processing  information  that  is  both  acquired 
from  the  environment  and  recalled  from  memory  to  make  judgments 
of  probable  cause.  There  are,  however,  significant  differences  in 
causal  versus  statistical  (or  normative)  reasoning. 

Kahneman,  Slovic,  and  Tversky  (1982)  have  compiled  an  anthology 
of  research  studies  demonstrating  that,  when  compared  to  the 
tenets  of  probability  and  statistical  theory,  humans  have  limited 
appreciation  for  the  concepts  of  randomness,  statistical 
independence,  sampling  variability,  data  reliability,  regression 
effects,  and  the  like,  when  making  probabilistic  judgments.  All 
of  these  concepts  are  important  when  making  the  types  of  causal 
inferences  inherent  in  intelligence.  Although  these  concepts  and 
cognitive  heuristics  can  be  represented  in  the  lens  model 
paradigm  of  SJT,  the  tasks  used  to  investigate  them  and,  in  turn, 
orientation  have  been  quite  different  in  cognitive  heuristics 
research.  We  first  overview  the  kinds  of  tasks  used  in  cognitive 
heuristics  research  and  then  consider  important  research 
findings. 


3. 2. 2.1  Task  Characteristics  -  The  typical  tasks  used 
in  cognitive  heuristics  research  are  simple  word  problems 
employing  quantitative  information  that  is  optimally  combined 
using  the  rules  of  probability  and  statistical  theory,  von 
Winter feldt  (1988,  p.  467)  has  described  the  paradigm  as  follows: 
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"(1)  A  formal  rule,  known  to  the  experimenter  but  not  known 
or  available  to  the  subject  is  applied  in  formulating 
an  intellectual  task; 

(2)  Subjects  are  asked  to  solve  the  task  intuitively 
(without  tools);  and 

(3)  A  systematic  discrepancy  between  the  formal  rule  and 
subjects'  answers  is  found." 


The  following  example  is  a  classic  problem  studied  by  Tversky  and 
Kahneman  (1980)  and  that  is  discussed  in  detail  in  Hogarth 
(1987)  : 


"A  cab  was  involved  in  a  hit-and-run  accident  at  night. 
Two  cab  companies,  the  Green  and  the  Blue,  operate  in 
the  city.  You  are  given  the  following  data; 

(1)  85%  of  the  cabs  in  the  city  are  Green  and  15%  are 
Blue. 

(2)  A  witness  identified  the  cab  as  a  Blue  cab.  The  court 
tested  his  ability  to  identify  cabs  under  the  appro¬ 
priate  visibility  conditions.  When  presented  with  a 
sample  of  cabs  (half  of  which  were  Blue  and  half  of 
Green) ,  the  witness  made  correct  identifications  in 
80%  of  the  cases  and  erred  in  20%  of  the  cases. 

Question;  What  is  the  probability  that  the  cab 
involved  in  the  accident  was  Blue  rather  than  Green?" 


The  typical  response  to  this  question  is  about  80%.  This  is 
consistent  with  the  evidence  that  the  witness  correctly 
identified  80%  of  the  cases  in  the  court’s  test  of  his  ability  to 
correctly  identify  cabs.  However,  the  correct  answer  to  the 
problem  is  that  the  chance  of  the  cab  being  Blue  is  41%.  The 
reason  so  many  people  are  so  far  from  the  correct  probability 
estimate,  and  even  the  correct  color  of  the  cab,  is  that  they 
often  fail  to  consider  the  "a  priori"  or  base-rate  information 


that  85%  of  the  cabs  in  the  city  are  Green,  not  Blue.  Probability 
theory,  and  in  particular  Bayes'  Theorm,  requires  that  one 
sytematically  use  both  the  base  rate  information  and  the  new 
evidence  when  making  a  probabilistic  inference.  Specifically, 


p(Blue)  p(Say  Blue|Blue)  (.15)  (.8) 

p(Blue|Say  Blue)  - - - -  =  .41 

p(Say  Blue)  .29 


where  p(Say  Blue)  =  p(Blue)  p(Say  Blue|Blue)  + 

p(Green)  p(Say  BluejGreen) 

=  (.15  X  .80)  +  (.85  X  .20)  =  .29 

It  is  important  to  note  that  when  the  same  base  rate  information 
is  given  a  causal  meaning  by  rephrasing  the  problem  to  say  "85% 
of  accidents  in  the  city  involve  Green  cabs  and  15%  involve  Blue 
cabs"  the  average  response  is  55%,  much  closer  to  that  which  is 
optimal  according  to  Bayes'  Theorem. 

3. 2. 2. 2  Important  Findings  -  The  above  example 
illustrates  that  people  give  meaning  to  information  in  order  to 
make  causal  sense  out  of  it.  Probability  theory,  however,  does 
not  necessarily  do  so,  for  probability  theory  is  merely  a  set  of 
rules  that  permit  one  to  infer  the  relationship  between 
probabilistic  events  if  certain  assumptions  are  met.  In  fact, 
Einhorn  and  Hogarth  (1986)  and  Hogarth  (1987)  have  argued  that 
the  nature  of  causal  reasoning  not  only  differs  in  important 
respects  from  the  dictates  of  probability  theory,  but  that 
certain  aspects  of  probability  theory  are  antithetical  to  causal 
reasoning.  As  they  point  out,  for  example,  causal  reasoning  is 


generally  unidirectional  (e.g.,  X  causes  Y) .  On  the  other  hand, 
in  statistical  logic  the  relation  between  two  events  can  be,  and 
often  is,  discussed  in  either  or  both  directions.  For  example,  in 
order  to  use  Bayes'  Theorem  to  calculate  the  posterior  probabil¬ 
ity  of  a  hypothesis  given  new  data  or  information  [i.e.,  P(H|D)], 
one  needs  to  assess  the  likelihood  of  the  data  given  the 
hypothesis  [i.e.,  P(D|H)];  and,  whereas  statistical  theory  is 
based  on  the  logical  structure  of  information,  causal  reasoning 
is  responsive  to  both  structure  and  content  in  terms  of  the 
causal  field,  cues-to-causality ,  and  the  plausibility  of 
alternative  scenarios  and  causal  explanations.  In  summary, 
although  it  may  not  be  normatively  correct  when  compared  to 
probability  and  statistical  theory,  people  use  heuristics  that 
weight  information  on  the  basis  of  its  perceived  causal  meaning 
not  its  statistical  diagnosticity . 

The  heuristics  that  humans  use  to  attach  meaning  to  information 
makes  us  susceptible  to  "biases,"  when  compared  to  some  normative 

(or  presumed  "optimal")  standard,  depending  on  how  the  problem  is 
presented  or  "framed"  (Tversky  and  Kahneman,  1981).  This  point 
was  illustrated  with  the  taxicab  problem  presented  above. 
Substantial  psychological  research  has  been  performed  trying  to 
identify  the  nature,  cause,  and  implications  of  these  biases.  In 
an  effort  to  synthesize  this  research,  Hogarth  (1987)  has 
catalogued  cognitive  biases  according  to  the  following  four 
information  processing  stages:  acquisition,  processing,  output, 
and  feedback.  In  the  following  subsections,  we  will  list  some  of 


83 


the  biases  that  seem  most  relevant  to  intelligence  (Table  3.1); 
those  that  are  most  relevant  to  operations  will  be  considered 
later  (Table  3.3). 

Before  doing  so  we  want  to  note  that  the  word  "bias"  has  a 
negative  connotation.  Indeed,  the  research  on  cognitive  biases 
has  often  been  presented  as  a  cataloguing  of  human  fallibilities. 
However,  the  cognitive  heuristics  that  spawn  these  biases  have 
both  strengths  and  weaknesses.  On  the  positive  side,  they  permit 
humans  with  limited  information  acquisition,  retention,  and 
processing  capabilities  to  not  only  establish  order  and  meaning 
out  of  the  mass  of  information  with  which  they  are  confronted, 
but  to  develop  new  and  creative  ways  of  ever  improving  (and 
hopefully,  never  destroying)  our  environment.  On  the  negative 
side,  they  expose  limitations  in  reasoning  when  compared  to 
normative  models  of  decision  behavior.  This  research  is 
particularly  important  to  the  design  of  advanced  information 
systems  because  it  suggests  that  how  information  is  presented  by 
these  systems  can  significantly  affect  the  adequacy  of  our 
thinking.  Moreover,  probability  and  statistical  theory  are  not 
esoteric  concepts;  they  provide  us  with  powerful  rules  for 
enhancing  causal  inference  under  conditions  of  uncertainty. 

These  rules  can  be  and  have  been  used  to  improve  unaided  human 
decision  behavior  (e.g.,  see  Kelly  et  al.,  1981).  The  long-term 
goal  of  cognitive  engineering  is  the  design  of  support  systems 
that  effectively  combine  human  and  computer  strengths  and, 
thereby,  improve  human  decision  behavior. 
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Table  3.1:  Cognitive  Heuristics  and  Biases  Relevant  to  Intelligence 


3.2.3  Biases  in  Information  Acquisition  -  Hogarth  (1987) 
has  suggested  that,  "the  issue  of  bias  in  information  acquisition 
can  be  conceptualized  by  enquiring  when  and  why  information 
becomes  salient  to  an  individual.  This  question  can  be  further 
broken  down  by  noting  that  information  can  be  accessed  from  two 
sources:  (1)  the  individual's  memory;  and  (2)  the  task 

environment."  First,  we  consider  cognitive  biases  that  are 
memory-based;  second,  we  consider  those  that  are  facilitated  by 
task  characteristics;  and  third,  we  consider  biases  that  are 
facilitated  by  the  way  that  data  is  presented,  which  is  a 
particularly  relevant  task  characteristic  for  display  technology. 

We  consider  three  memory-based  biases  that  affect  data 
acquisition.  The  first  is  the  availability  bias  (e.g.,  see 
Tversky  and  Kahneman,  1973).  Specifically,  the  ease  with  which 
specific  instances  can  be  recalled  from  memory  affects  judgments 
of  the  frequency  of  different  events.  In  particular,  publicity  or 
extensive  discussion  of  (or  focus  on)  particular  events,  makes 
them  more  salient  and,  in  turn,  more  av  ilable  in  memory.  The 
second  bias  is  selective  perception  (e.g.,  see  Dearborn  and 
Simon,  1958).  In  particular,  people  structure  problems  on  the 
basis  of  their  own  experience  such  that  anticipations  of  what  one 
expects  sometimes  bias  what  one  does  see.  The  third  bias,  which 
is  related  to  selective  perception,  is  the  confirmation  bias 
(e.g.,  see  Wason,  1960).  In  particular,  people  e jek  information 
consistent  with  their  own  views  and  hypotheses  instead  of  seeking 
disconf irming  information.  Of  particular  interest  is  a  recent  set 
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of  experiments  by  Tolcott  and  his  colleagues  (1989a,  1989b) 
demonstrating  the  confirmation  bias  with  Army  tactical 
intelligence  analysts. 

We  now  consider  two  task-based  biases  affecting  memory 
acquisition.  The  first  is  the  frequency  bias.  Specifically, 
people  often  judge  the  strength  of  predictive  relations  by 
focusing  on  the  observed  frequency  of  events  rather  than  their 
observed  relative  frequency.  As  Einhorn  and  Hogarth  (1978)  have 
shown,  information  on  the  non-occurrence  of  an  event  is  often 
unavailabe  and  frequently  ignored  when  available.  Second, 
concrete  information,  that  is,  information  that  is  vivid  or  based 
on  experience  or  incidents,  dominates  abstract  information,  such 
as  summaries  and  statistical  base-rates.  According  to  Nisbett  and 
Ross  (1980),  concrete  and  vivid  information  contributes  to  the 
" imaginability"  of  the  information  and,  in  turn,  enhances  its 
impact  on  inference  and  decision-making. 

There  is  a  substantial  amount  of  research  suggesting  that  the  way 
the  same  information  is  presented  can  significantly  affect 
information  acquisition.  For  example,  research  by  Einhorn  and 
Hogarth  (1987)  and  Serfaty  et  al.  (1988),  the  latter  with 
experienced  Army  personnel,  showed  that  the  same  information 
disconf irming  an  initial  hypothesis  had  a  significantly  greater 
impact  when  it  was  presented  at  the  end  versus  the  beginning  of  a 
series.  Russo  (1977)  has  shown  that  intact  displays  containing 
all  available  information  result  in  better  performance  than  the 
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sequential  presentation  of  the  same  information.  Yet,  intact 
displays  are  not  without  their  potential  pitfalls.  Fischhoff  et 
al.  (1978),  for  example,  demonstrated  that  seemingly  complete 
presentations  of  information  via  logical  displays  can  blind 
experts,  in  this  case  auto  mechanics,  to  critical  omissions  in 
the  data.  "All  information  presented  seems  so  consistent  that 
the  individual  is  only  able  to  come  to  one,  possibly  erroneous 
conclusion"  (Hogarth,  1987).  Such  an  effect  is  consistent  with 
the  confirmation  bias-  More  generally,  the  way  information  is 
presented  affects  its  saliency  and,  in  turn,  the  information  that 
is  acquired  to  make  causal  sense  out  of  the  world. 

3.2.4  Biases  in  Information  Processing  -  Related  to 
information  acquisition  is  its  processing;  that  is,  the 
heuristics,  rules,  or  more  generally,  ways  that  different  pieces 
of  information  are  combined  into  an  inference  or  decision. 
Consistent  with  the  paradigm  for  cognitive  heuristics  research, 
the  literature  is  replete  with  processing  biases  compared  with 
normative  combination  rules.  Some  of  the  biases  most  relevant  to 
intelligence  analysis  and  production  are  listed  here. 

Humans  are  conservative  information  processors  compared  to  Bayes' 
Theorem  (e.g.,  see  Edwards,  1968).  Even  when  the  task  is 
structured  to  make  us  focus  on  base  rates,  we  do  not  revise  our 
opinions  on  the  receipt  of  new  information  as  much  as  we  should 
compared  to  Bayes'  Theorem.  This  is  not  surprising,  for  Bayes' 
Theorem  is  neither  intuitively  obvious  nor  a  trivial  analytical 
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calculation.  Instead,  we  appear  to  use  two  very  simple  heuristics 
to  revise  probability  estimates  on  the  basis  of  new  data.  First, 
there  is  the  representativeness  heuristic  (e.g.,  see  Kahneman  and 
Tversky,  1973),  which  was  illustrated  in  the  cab  example.  That 
is,  we  sometimes  judge  the  likelihood  of  events  by  estimating 
their  similarity  to  the  class  of  events  of  which  they  are  suppose 
to  be  an  exemplar,  such  as  the  witness'  80%  accuracy  in  the  cab 
example.  The  second  heuristic  is  anchoring  and  adjustment. 
Predictions  are  made  by  anchoring  on  a  cue  or  value  and  then 
adjusting  to  allow  for  the  circumstances  of  the  present  case. 
Einhorn  and  Hogarth  (1985,  1987)  have  presented  empirical 
findings  supporting  the  use  of  anchoring  and  adjustment  when 
people  are  forced  to  make  probability  estimates  in  the  face  of 
highly  ambiguous  data,  and  to  explain  the  effect  of  information 
presentation  order  on  inference,  respectively. 

A  number  of  other  processing  biases  relevant  to  causal  inference 
have  been  identified  in  the  literature.  For  example,  there  is  the 
conjunction  fallacy  (e.g.,  Slovic  et  al.,  1976).  In  probability 
theory,  the  joint  probability  of  two  events  cannot  be  larger  than 
the  probability  of  the  smaller  of  the  two  events  (e.g.,  .1  x  1.0 
=  .1).  Yet,  substantial  research  shows  that  when  word  problems 
are  framed  to  elicit  causal  reasoning  instead  of  statistical 
reasoning,  people  estimate  the  joint  probability  of  two  events  as 
being  larger,  not  smaller,  than  the  probability  of  the  smaller 
event.  In  one  experiment  of  potential  interest  to  intelligence 
analysts,  Tversky  and  Kahneman  (1983)  asked  (in  1982) 
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professional  forecasters  to  assess  the  probability  of  "a  complete 
suspension  of  diplomatic  relations  between  the  USA  and  the  Soviet 
Union  sometime  in  1983."  A  second  group  of  forecasters  was  asked 
to  evaluate  this  outcome  and  "a  Russian  invasion  of  Poland."  The 
second  group's  probabilities  were  higher  than  the  first's, 
thereby  violating  the  laws  of  probability  theory. 

SJT  research  informs  us  that  people  are  olten  inconsistent  in  how 
they  process  information.  In  adoition,  we  are  subject  to  what 
Tver sky  and  Kahneman  (1971)  have  called  the  "law  of  small 
numbers"  in  ''ontrast  to  the  Law  of  Large  Numbers.  In  particular, 
problems  can  be  framed  in  such  a  way  that  people,  including 
trained  scientists,  can  give  undue  confidence  to  a  (relatively) 
small  amount  of  data  —  and  we  are  subject  to  regression  biases. 
According  to  Hogarth  (1987),  "...  extreme  values  of  cues  [without 
perfect  predictability]  (e.g.,  test  scores)  are  typically 
accompanied  by  less  extreme  values  of  the  criterion  (e.g., 
performance  measures).  Thus,  a  sensible  judgmental  strategy  is 
to  regress  predictions  based  on  extreme  observations  toward  the 
mean  of  the  variable  predicted.  However,  people  frequently  make 
implicit  use  of  extreme  values  of  predictive  cues,  together  with 
consistency  of  data  sources,  to  justify  confidence  in  their 
predictions.  Paradoxically,  characteristics  of  information  that 
inspire  confidence  are  often  inversely  related  to  the  predictive 
accuracy  of  that  information.  This  has  led  to  what  Kahneman  and 
Tversky  [1973]  have  termed  the  "illusion  of  validity." 


In  rlosing  this  subsection,  it  is  important  to  note  that  task 
characteristics  can  significantly  affect  causal  inference.  In 
particular,  Payne  et  al.  (1987)  has  shown  that  time  pressure 
significantly  affects  the  amount  of  information  examined  and  the 
rules  used  to  process  it.  Janis*  (1972)  classic  work  on 
"groupthink"  shows  how  social  pressure  within  highly  cohesive, 
strongly  directed  groups  can  unduly  influence  inferences.  As 
Janis  and  Mann  (1977)  point  out,  “when  the  degree  of  complexity 
of  an  issue  exceeds  the  limits  of  cognitive  abilities,  there  is  a 
marked  decrease  in  adequacy  of  information  processing  as  a  direct 
effect  of  information  overload  and  ensuing  fatigue." 

3.2.5  Biases  Due  to  Output  or  Feedback  -  As  Hogarth  (1987) 
suggests,  "output  biases  appear  to  be  triggered  by  the  way  in 
which  people  express  judgement  or  choice  .  .  .  the  importance  of 
feedback  in  judgement  relates  to  its  effect  on  learning.” 
Regarding  the  former,  Hogarth  (1975)  has  shown,  for  example,  that 
probability  estimates  can  depend  on  how  people  have  been  asked  to 
respond  and  on  the  scale  used  to  measure  these  responses.  This 
output  bias  is  so  strong  that  Spetzler  and  Stael  von  Holstein 
(1975)  strongly  recommended  that  decision  analysts  use  multiple 
methods  to  converge  on  probabilistic  assessments  of  uncertainty. 
The  reader  should  not  think  that  this  output  bias  is  simply  a 
function  of  using  numbers  instead  of  words  to  represent 
uncertainty.  Moore  (1977)  found  substantial  differences  in  how 
experienced  managers  ranked  expressions  used  to  express 
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uncertainty.  In  intelligence,  Kent  (1964)  "...  was  so  concerned 
with  the  fact  that  almost  all  intelligence  analysis  documents 
contained  ambiguous  verbal  reports  of  uncertainty,  that  he 
proposed  a  set  of  rules  for  translating  words  into  probabilities 
and  vice  versa  ..."  (von  Winterfeldt,  1988).  More  generally,  as 
Fischhoff  et  al.  (1980)  have  shown,  the  way  a  person  is  asked  to 
respond  can  significantly  affect  his  judgment. 

As  the  SJT  research  indicates,  outcome  feedback  does  not 
necessarily  result  in  high  levels  of  performance.  Part  of  the 
problem  is  that  inference  tasks  are  seldom  perfectly  predictable 
(i.e.,  Re  §  1);  consequently,  outcome  feedback  is  seldom  of 
perfect  validity.  In  addition,  as  Einhorn  and  Hogarth  (1978)  have 
pointed  out,  it  is  often  not  possible  to  observe  outcomes 
associated  with  the  total  range  of  judgments,  and  feedback  is 
often  delayed  or  affected  by  other  events.  In  particular,  recent 
research  on  dynamic  decision  making  (e.g.,  see  Sterman,  1989)  has 
shown  that  delays  in  receiving  outcome  feedback  can  result  in 
misperceptions  that  essentially  make  people  insensitive  to  it. 

Such  settings  have  been  referred  to  as  "outcome  irrelevant 
learning  structures"  by  Hogarth  (1987). 

In  addition  to  task  characteristics,  people  are  not  always 
effective  in  using  feedback.  For  example,  people  often 
fflisperceive  chance  fluctations  and,  in  some  cases,  erroneously 
attribute  causation  to  chance  events.  A  good  example  is  the 
"gambler's  fallacy"  where  people  expect  a  chance  event  (e.g.. 


Black  in  roulette)  because  they  have  just  observed  a  large  number 
of  unexpected  (but  chance)  events  (e.g.,  a  number  of  Reds).  In 
addition,  people  cannot  remember  all  the  feedback  they  receive. 
This  can  result  in  logical  fallacies  in  recall.  In  studying 
eyewitness  testimony,  Loftus  (1979)  has  shown  that  recall  can  be 
influenced  by  post-event  information  and  the  way  information  is 
elicited  from  the  witness.  This  appears  to  occur  because  memory 
is  based  on  the  reconstruction  of  events  to  make  causal  sense  out 
of  the  world.  Hogarth  (1987)  suggests  that  "...  the  paradox 
of  memory  is  that  long-term  memory  does  not  work  by  remembering 
what  is  actually  recalled,  but  rather  by  remembering  fragments  of 
information  that  allow  one  to  construct  more  complete 
representations  of  the  information."  Seen  from  this  perspective, 
the  hindsight  bias,  which  is  colloquially  referred  to  as  Monday 
morning  quarterbacking  or  second-guessing,  is  not  surprising. 
Perhaps  what  is  surprising,  is  the  strength  tor,  as  Fischhoff 
(1975)  found,  people  appear  adept  at  interpreting  new  information 
as  consistent  with  previously  held  positions. 

3.2.6  Analogical  Reasoning  -  Analogical  —  or  "case-based" 
—  reasoning  is  a  knowledge  representation  and  control 
methodology  which  can  assist  planners  and  designers  in  making 
complex,  domain-specific  decisions  or  problem  assessments  and 
recommendations  based  upon  previous  experiences  and  patterns  of 
previous  experiences.  These  previous  experiences,  or  "cases,"  of 
domain-specific  knowledge  and  action,  are  used  in  comparison  with 
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new  situations  or  problems;  and  these  past  methods  of  solution 
provide  expertise  for  use  in  those  new  situations  or  problems  the 
system  is  built  to  handle.  Schank  and  Abelson  (1977),  Kolodner, 

Simpson  and  Sycara-Cyranski  (1985),  among  others,  have  examined 
the  applicability  of  developing  automated  systems  for  reasoning 
based  upon  previous  experience. 

Reasoning  by  analogy  is  a  natural  process.  Analogical  reasoning 
can  occur  in  every  situation  in  which  people  are  required  to  make 
judgments  and  predictions.  These  situations  may  be  highly 
technical  ones  or  be  of  a  more  mundane  nature.  A  very  simple 
example  of  the  use  of  analogical  reasoning  may  be  seen  when 
someone  buys  or  sells  a  home.  The  realtor  sets  a  price  for  a 
property  not  by  using  a  formal  model  and  calculating  all  of  the 
variables,  but  by  choosing  a  comparable  sale  and  adjusting  the 
price  on  the  basis  of  small  differences  between  the  two 
properties  (such  as  an  extra  bathroom  or  a  location  on  a  corner 
lot).  Engineers  have  traditionally  made  use  of  analogies  in 
prediction  and  design,  typically  by  looking  for  structural 
comparisons.  Intelligence  analysts  also  reason  analogically. 

Previous  cases  influence  interpretations  of  current  events. 

This  concludes  our  brief  overview  of  cognitive  science  research 
on  inference  that  may  be  applicable  to  the  cognitive  engineering 
of  advanced  information  and  decision  systems  for  intelligence.  We 
now  consider  research  findings  that  may  be  applicable  to 
operations  and  subsequently,  to  facilitating  the  communication 
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between  intelligence  and  operations.  We  again  caution  the  reader 
that  our  overview  is  small  and  selective.  Our  goal  is  not  to 
survey  all  potentially  applicable  areas  of  research.  Rather,  our 
goal  is  to  commence  the  process  of  mapping  computer  technology  to 
coanitive  processes. 

3.3  Operational  Decision  Making 

This  section  overviews  the  following  areas  of  cognitive  research, 
all  of  which  are  applicable  to  the  option  generation,  evaluation, 
and  selection  processes  inherent  in  operations: 

•  Bounded  rationality, 

•  Prospect  theory,  and 

•  Cognitive  heuristics  involving  decision  (versus 
inference)  processes. 

3.3.1  Bounded  Rationality  -  The  concept  of  bounded 
rationality  is  attributed  to  Nobel  laureate  Herbert  Simon  (e.g., 
see  1955,  1979;  also  Hogarth,  1987,  and  March,  1978,  for  general 
discussions),  who  argued  that  humans  lack  both  the  knowledge  and 
computational  skill  required  to  make  decisions  in  a  manner 
compatible  with  economic  notions  of  rational  behavior.  In  order 
to  deal  with  human  and  task  limitations  or  bounds,  Simon  argued 
that  humans  simplified  decision  problems  so  that  they  can  address 
them  in  a  "reasonable"  if  not  economically  "rational"  manner.  In 
particular,  Simon's  approach  was  to  specify  what  the  rational 
economic  model  required  humans  to  know  and  do,  and  then  ask  how 


they  could  cope  with  the  task  given  their  limited  knowledge, 
memory,  and  processing  capabilities.  Therefore,  the  approach  is 
conceptually  similar  to  that  described  above  for  cognitive 
heuristics  research. 

3. 3. 1.1  Task  Characteristics  -  The  rational  model’s 

requirements  are  illustrated  by  the  concept  of  a  payoff  matrix, 

an  example  of  which  is  presented  in  Table  3.2.  The  rows  of  the 

matrix  represent  all  of  the  different  alternatives  available  to 

the  decision  maker  for  solving  a  particular  decision  problem. 

The  columns  represent  all  of  the  different  states  of  the  world, 

as  defined  by  future  events,  that  could  affect  the  attractiveness 

of  the  alternatives.  The  p  ...p  values  represent  the 

1  k 

probabilities  for  each  state  of  the  world.  The  cell  entries  in 
the  matrix  indicate  the  value  (or  "utility")  of  the  outcome  or 
"payoff"  for  each  combination  of  alternatives  and  states  of  the 
world.  Each  outcome  is  presumed  to  represent  a  cumulative  payoff 
comprised  of  perceived  advantages  and  disadvantages  on  multiple 
criteria  of  varying  importance  to  the  decision-maker.  Finally, 
the  rational  decision  maker  is  required  to  select  the  alternative 
that  maximizes  expected  utility,  which  is  calculated  for  each 
alternative  by  first  multiplying  the  values  for  the  outcomes  and 
the  probabilities  for  the  states  of  nature,  and  then  summing  the 
products. 
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STATES  OF  NATURE 


Table  3.2:  Rational  Economic  Model  In  a  Payoff  Matrix 


3. 3. 1.2  Important  Findings  -  The  rational  economic 
model  clearly  assumes  that  the  decision-maker  has  extensive 
knowledge  and  impressive  unaided,  computational  power.  In 
addition  to  Simon's  research,  substantial  psychological  research 
(e.g.,  see  the  reviews  by  Einhorn  and  Hogarth,  1981;  Slovic,  et 
al.  1977)  indicates  the  inadequacy  of  these  (and  other) 
assumptions  of  the  model.  Therefore,  how  do  we  cope  with  the 
cognitive  demands  represented  in  the  decision  matrix?  How  does 
unaided  human  decision  behavior  remain  purposeful  and 
"reasonable"  given  the  dynamic  nature  of  the  environment  and  our 
inherent  information  acquisition  and  processing  limitations? 

Simon  suggested  three  simplification  strategies.  First,  people 
simplify  the  decision  problem  by  only  considering  a  small  number 
of  alternatives  and  states  of  nature  at  a  time.  Second,  people 
simplify  the  evaluation  problem  by  setting  aspiration  (or 
acceptability)  levels  on  the  outcomes;  and,  third,  people 
simplify  the  selection  problem  by  choosing  the  first  alternative 
that  satisfies  the  aspiration  level.  In  other  words,  people  do 
not  optimize  (i.e.,  choose  the  best  of  all  possible 
alternatives),  but  satisfice  (i.e.,  choose  the  first  satisfactory 
alternative).  In  this  way,  people  can  reduce  information 
acquisition  and  processing  demands  and  still  act  in  a  purposeful, 
reasonable  manner. 

The  strategies  in  Simon's  theory  of  bounded  rationality  are  not, 
however,  without  their  costs.  First,  as  Hogarth  (1987)  has 


pointed  out,  research  on  creativity  suggests  that  one  of  the 
biggest  deficiences  in  human  decision  behavior  is  our  failure  to 
sufficiently  imagine  the  range  of  alternatives  at  our  disposal 
and  the  various  events  that  could  occur  in  the  future.  Second, 
aspiration  levels  may  be  unrealistically  high  or  low.  The  former 
could  well  result  not  only  in  the  elimination  cf  potentially  good 
alternatives  early  in  the  decision  process,  but  the  acceptance  of 
a  relatively  inferior  alternative  later  in  the  process  because 
subsequent  events  have  forced  us  to  lower  our  aspiration  level. 

In  contrast,  unrealistically  low  aspiration  levels  and  the 
satisficing  strategy  may  well  result  in  the  acceptance  of 
relatively  poor  alternatives  early  in  the  decision  process. 

It  is  important  to  emphasize  that  bounded  rationality  represents 
a  descriptive  theory  of  human  decision  behavior.  It  does  not 
specify  how  people  should  make  decisions  but,  rather,  presents  a 
theoretical  perspective  on  how  people  do  make  decisions  given  a 
complex,  dynamic  environment  and  limited  information  acquisition 
and  processing  capabilities.  Moreover,  subsequent  research 
indicates  that  people  are  quite  capable  of  using  other,  in  some 
cases  more  complex  strategies  than  the  three  proposed  by  Simon. 

In  contrast,  the  rational  economic  model  is  now  typically  seen  as 
a  prescriptive  not  descriptive  theory  of  decision  making.  It  is 
typically  referred  to  as  decision  theory  (or  expected  utility 
theory  or  subjective  expective  utility  theory) ,  and  it  provides 
an  axiomatic  basis  for  specifying  how  people  should  make 
decisions,  as  represented  in  the  decision  matrix,  given  that  they 


95 


accept  certain  logically  defined  principles  of  behavior. 

Moreover,  analytical  procedures  called  decision  analysis  and 
various  support  systems  have  been  developed  to  help  people 
implement  decision  theory.  Although  there  are  many  books  on 
decision  theory  and  decision  analysis,  the  texts  by  Brown  et  al. 
(1974),  Watson  and  Buede  (1987),  von  Winter feldt  and 
Edwards  (1986),  among  others,  represent  good  introductions. 

3.3.2  Prospect  Theory  -  Kahneman  and  Tver sky  (1979)  have 
proposed  "prospect  theory"  as  a  descriptive  theory  of  choice 
under  uncertainty.  Like  Simon’s  bounded  rationality,  prospect 
theory  is  juxtaposed  against  expected  utility  theory.  What  is 
particularly  important  about  prospect  theory  for  cognitive 
engineering  is  that  it  distinguishes  between  two  phases  in  the 
choice  process.  The  first  phase  is  called  editing;  its  purpose 
is  to  simplify  the  presented  information  in  the  choice  setting  in 
order  to  enhance  decision  making.  The  second  phase  is  called 
evaluation;  its  purpose  is  to  analyze  the  edited  choices  (i.e., 
prospects)  so  that  the  decision  maker  can  select  the  one  with  the 
highest  personal  value.  What  Kahneman  and  Tversky  have  shown  is 
that  the  way  the  prospect  information  is  presented  to  people 
significantly  affects  how  they  edit  and  evaluate  it  such  that 
information  that  should  result  in  the  same  choice  from  the 
perspective  of  expected  utility  theory  actually  results  in 
different  choices. 


100 


3. 3. 2.1  Task  Characteristics  -  Subjects  in  the 


experiments  used  to  generate  prospect  theory  are  faced  with 
simple  prospects  (or  choices)  that  have  a  correct  answer  based  on 
expected  utility  theory.  For  example,  the  following  prospect  is 
taken  from  Kahneman  and  Tversky  (1979): 

Choice  A:  ($4000  with  p  =  .8;  $0  with  p  =  .2)  or 
Choice  B:  ($3000  for  sure;  that  is,  p  =  1.0) 

The  majority  of  participants  will  select  Choice  B.  Yet,  Choice  A 
has  the  greater  expected  value;  that  is,  $4000  x  .8  =  3200).  Now, 
consider  the  following  prospect: 

Choice  C:  (-$4000  with  p  =  .8;  $0  with  p=  .2)  or 
Choice  D:  (-$3000  for  sure;  that  is,  p  =  1.0). 

The  only  change  in  the  second  prospect  is  that  the  sign  has  been 
reversed  so  that  one  is  now  considering  losses,  not  gains. 

However,  in  this  case,  the  majority  of  the  subjects  picked  Choice 
C.  That  is,  they  would  now  be  willing  to  take  a  gamble  instead  of 
losing  $4000  with  a  probability  of  .8  instead  of  taking  a  sure 
loss  of  $3000.  Again,  they  have  selected  the  choice  with  the 
lower  expected  value. 

Numerous  other  examples  have  been  used  to  demonstrate  that 
people’s  choices  are  not  always  consistent  with  expected  utility 
theory.  More  generally,  Hogarth  (1987)  has  pointed-out  tha*- ,  by 
and  large,  the  research  has  been  directed  toward  testing  the 
general  principles  of  decision  theory,  which  assume  that  people 
express  consistent  beliefs  in  the  form  of  predictive  judgments 
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and  consistent  preferences  in  the  form  of  evaluative  judgments. 

The  principles  regarding  preferences  include  transitivity  (if  A 
is  prefered  to  B  and  B  to  C,  then  A  should  be  preferred  to  C); 
dominance  (if  alternative  A  is  preferred  to  alternative  B  on  all 
dimensions,  then  there  should  be  no  way  that,  in  total,  B  should 
be  preferred  to  A);  and  invariance  (one's  preference  for  two 
options  should  not  be  affected  by  the  way  one  presents 
information  about  them).  "Perhaps  the  most  striking  feature  of 
these  principles  is  that,  whereas  they  are  accepted  as  reasonable 
when  stated  in  the  abstract  form,  their  implications  are  often 
violated  in  actual  choices"  (Hogarth,  1987).  Prospect  theory 
provides  some  insights  as  to  why  this  occurs. 

3. 3. 2. 2  Important  Findings  -  Kahneman  and  Tver sky 
(1979)  have  hypothesized  that  the  editing  and  evaluation  phases 
that  people  use  to  make  decisions  under  risk  have  distinct 
operations.  For  example,  Kahneman  and  Tversky  proposed  six  editing 
operations:  coding,  combination,  segregation,  cancellation, 

simplification,  and  dominance.  With  respect  to  "coding,"  people 
perceive  outcomes  as  gains  or  losses  from  a  referent  point  rather 
than  final  states  (e.g.,  of  wealth).  The  current  position  is 
usually  considered  as  the  referent  point.  However,  the  location 
of  the  reference  point  and,  in  turn,  the  coding  of  outcomes  as 
gains  or  losses,  can  be  affected  by  how  the  prospects  are 
formulated,  as  well  as  by  the  person's  expectations.  This  coding 
is  particularly  important  in  framing  decisions  because,  as  the 
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example  presented  above  indicates,  people  tend  to  be  risk  adverse 
when  considering  gains  and  risk  seeking  when  considering  losses, 
particularly  il  one  cf  the  prospects  is  certain.  The  research  by 
Kahneman  and  Tversky  (1979,  1981)  and  others  (e.g.,  McNeil  et 
al.,  1982;  Payne  et  al.,  1980)  indicates  that  (1)  reference 
points  can  be  manipulated,  and  (2)  losses  loom  much  larger  than 
gains . 

The  other  five  editing  operations  are  used  to  simplify  the 
choice.  For  example,  a  prospect  can  be  simplified  by  combining 
the  probabilities  associated  with  the  same  outcomes.  However, 
this  can  sometimes  result  in  inappropriate  problem  representa¬ 
tions  because,  as  the  cognitive  heuristics  research 
indicates,  people  do  not  always  implement  probability  theory 
correctly.  Similarly,  people  appear  to  segregate  the  riskless 
from  the  riskless  components  of  prospects  (called  segregation) 
and  cancel-out  aspects  shared  in  common  between  prospects  (called 
cancellation).  Although  quite  reasonable  operations  for 
simplifying  complex  problems,  they  can  result  in  different 
choices  simply  depending  on  how  the  problem  is  framed  (e.g.,  see 
Hogarth,  1987). 

Kahneman  and  Tversky  have  proposed  two  operations  in  the 
evaluation  phase  of  prospect  theory;  a  value  function,  which  is 
analogous  to  the  utility  function  in  expected  utility  theory,  and 
a  decision  weight  function,  which  indicates  the  subjective 
importance  of  the  probabilities  in  prospects.  The  value  function 
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codes  the  psychological  value  of  gains  and  losses  from  the 
"coded"  reference  point.  The  function  is  steeper  for  losses  than 
for  gains,  consistent  with  the  ooservation  that  losses  loom  much 
larger  than  gains.  We  tried  to  capture  this  with  the  reversal 
illustrated  in  the  above  example.  Moreover,  outcomes  near  the 
reference  point  are  given  more  value  per  unit  change  than  units 
farther  from  the  reference  point.  For  example,  assuming  a 
reference  point  of  $0  and  a  range  from  $0  to  $1600,  a  $1 
difference  between  $0  and  $10  typically  has  more  psychological 
value  than  a  $1  difference  between  $1550  and  $1560. 

The  decision  weight  function  in  prospect  theory  links  the 
probabilities  in  prospects  to  choice.  In  particular,  the  function 
represents  the  finding  that  people  seem  to  overweight  low 
probabilities  and  underweight  high  probabilities  when  compared  to 
expected  utility  theory.  Interestingly,  prospect  theory  only 
defines  the  decision  weight  function  between  probabilities  of  0 
and  1.  These  two  probabilities  are  given  weights  of  0  and  1, 
respectively,  indicating  the  special  effect  of  certainty.  We  also 
tried  to  capture  this  in  our  example. 

3.3.3  Cognitive  Heuristics  -  We  now  briefly  consider 
cognitive  heuristics  and  biases  that  affect  option  evaluation, 
not  inference.  Of  course,  many  of  the  biases  considered  earlier 
in  this  review,  particularly  those  affecting  data  presentation, 
can  affect  option  evaluation  as  well  as  inference.  However,  our 
emphasis  here  is  on  optsrations  instead  of  intel  igence; 
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consequently,  we  will  only  consider  biases  that  are  predominantly 
oriented  to  option  evaluation,  not  inference.  (Cognitive  biases 
that  may  affect  both  intelligence  and  operations  are  listed  in 
Table  3.3  for  purposes  of  closure.) 

Cognitive  biases  affecting  evaluation  more  than  inference  are 
principally  found  in  the  information  processing  stages  of 
"output"  and  "feedback"  (Hogarth,  1987).  Considering  the  output 
stage  first,  we  previously  noted  that  the  manner  in  which  a 
person  is  required  to  respond  can  induce  bias.  For  example,  the 
relative  preference  for  gambles  can  be  reversed  when  people  are 
asked  to  express  choices  in  different  ways.  Moreover,  such 
preference  reversals  have  even  been  demonstrated  in  a  Las  Vegas 
gambling  casino  (Lichtenstein  and  Slovic,  1973)  and,  more 
recently,  with  computer-based  information  displays  (Johnson,  et 
al.,  1988).  In  addition,  people  engage  in  wishful  thinking. 

That  is,  contrary  to  expected  utility  theory,  people's  beliefs 
and  preferences  are  not  always  independent. 

As  we  also  previously  noted,  feedback  concerning  the  outcomes  of 
our  judgment  can  induce  bias  too.  This  bias  might  occur  either 
as  a  result  of  (1)  how  we  interpret  outcomes  or  (2)  the  nature  of 
the  environment  itself.  For  example,  regarding  the  former,  people 
have  a  tendency  to  attribute  success  to  their  skill  and  failure 
to  either  chance  or  the  situation  with  which  they  were  faced. 
Ironically,  people  tend  to  attribute  other  people's  failure's  to 
personality  traits,  not  the  situation.  Such  success/failure 
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•  In  "Information  Acquisition" 

•  Availability  of  Instances  in  Memory 

•  Selective  Perception  of  Information 

•  Focus  on  Confirming  (vs.  Oisconflrming)  Information 


Table  3.3:  Cognitive  Heuristics  and  Biases  Relevant  to  Operations 


attributions  appear  to  be  part  of  a  more  "fundamental  attribution 
error"  (e.g.,  see  Nisbett  and  Ross,  1980)  where  people  tend  "..to 
ignore  powerful  situational  determinants  of  behavior." 

Considering  the  latter,  outcomes  often  yield  inaccurate  or 
incomplete  information  concerning  predictive  relations.  As 
Einhorn  and  Hogarth  (1978)  have  discussed  in  detail,  in  most 
decision  making  settings  one  can  seldom  (if  ever)  learn  how  good 
one's  judgment  is  (e.g.,  as  operationalized  statistically  by  a 
correlation)  because  our  decisions  and  subsequent  actions  make  it 
impossible  to  know  what  the  effect  of  other  decisions  and  actions 
would  have  been. 

This  concludes  our  brief  overview  of  cognitive  science  research 
on  option  generation,  evaluation,  and  selection  that  may  be 
applicable  to  the  cognitive  engineering  of  advanced  information 
and  decision  systems  for  operations.  We  now  consider  research 
findings  that  may  be  applicable  to  facilitating  communication  of 
cognitive  processes  between  intelligence  and  operations.  In 
particular,  we  take  the  perspective  that  strategic  decision¬ 
making  can  be  facilitated  by  using  computer  technology  to 
represent  and  communicate  the  cognitive  processes  of  intelligence 
and  operations  personnel. 

3.4  Toward  Facilitating  the  Communication  of  Cognitive  Processes 
Between  Intelligence  and  Operations 

This  section  of  the  review  has  three  subsections.  First,  we 
present  three  decision  making  paradigms,  including  one  explicitly 
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developed  to  represent  military  decision-making,  that 
incorporates  situation  assessment  and  subsequent  decision-making 
as  part  of  decision-makers’  problem-solving  behavior.  These 
paradigms  provide  theoretical  support  for  the  importance  of 
trying  to  improve  the  communication  of  cognitive  processes 
between  intelligence  and  operations.  Second,  we  present  some 
findings  from  problem-solving  research  indicating  a  strong 
relationship  between  problem  definition  (i.e.,  situation 
assessment)  and  option  generation,  evaluation,  and  selection. 

This  research  provides  empirical  support  for  trying  to  improve 
the  communication  of  cognitive  processes.  Third,  we  overview 
some  SJT  research  demonstrating  that  computer  technology  can,  in 
fact,  be  successfully  employed  to  communicate  inference  processes 
and,  thus,  improve  interpersonal  understanding  and  decision¬ 
making.  This  position  is  also  supported  by  the  use  of  computer 
technology  to  support  group  decision-making  in  decision 
conferences,  an  area  of  applied  research  that  will  also  be  briefly 
considered. 

3.4.1  Decision-Making  Paradigms  -  Simon  (1960)  has  used 
three  categories  to  describe  decision  making  activities: 
intelligence,  design,  and  choice.  "Intelligence"  refers  to  the 
activities  inherent  in  problem  identification,  definition,  and 
diagnosis.  It  is,  as  Huber  (1980)  points  out,  the  conscious 
process  of  trying  to  explore  the  problem  in  an  effort  to  find-out 
the  current  state  of  affairs,  and  why  it  does  not  match  our 


desires.  "Design"  refers  to  those  activities  inherent  in 
generating  alternative  solutions  or  options  to  solving  the 
problem.  It  involves  "...  identifying  items  or  actions  that 
could  reduce  or  eliminate  the  difference  between  the  actual 
situation  and  the  desired  situation"  (Huber,  1980).  "Choice" 
refers  to  those  activities  inherent  in  evaluating  and  selecting 
from  the  alternatives.  It  is  the  action  that  most  people  think  of 
when  one  makes  a  decision. 

As  Huber  (1980)  and  others  (e.g.,  Andriole,  1989b;  Sage,  1986; 
Wohl,  1981)  have  pointed-out,  decision-making  activities  are  a 
subset  of  problem-solving  activities.  For  example,  the  first 
three  steps  in  Huber’s  five-step  problem-solving  paradigm  are 
those  activities  that  require  (X)  problem  identification, 
definition,  and  diagnosis;  (2)  the  generation  of  alternative 
solutions:  and  (3)  evaluation  and  choice  among  alternatives. 

These  steps  are  conceputually  identical  to  Simon's  decision¬ 
making  categories.  The  fourth  step  in  Huber's  paradigm  involves 
activities  inherent  in  implementing  the  chosen  alternative.  The 
fifth  step  involves  activities  inherent  in  monitoring  the 
implemented  action  in  an  effort  "...  to  see  that  what  actually 
happens  is  what  was  intended  to  happen"  (Huber,  1980).  If 
there  is  a  significant  mismatch  between  the  actual  and  desired 
state  of  affairs,  one  returns  to  step  #1,  exploring  the  problem. 

Wohl  (1981)  has  presented  a  paradigm  within  the  context  of 
military  tactical  decision-making  that  expands  on  the  activities 
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in  Simon's  and  Huber's  frameworks.  The  anatomy  of  Wohl's  SHOR 
(Stimulus  -  Hypothesis  Option  -  Response)  paradigm  is  presented 
in  Figure  3.2.  Intelligence  activities  are  differentiated 
between  the  Stimulus  and  Hypothesis  elements  of  the  SHOR 
paradigm.  In  particular,  the  Stimulus  element  is  comprised  of 
data  collection,  correlation,  aggregation,  and  recall  activities; 
it  naturally  includes  many  of  the  activities  also  included  in 
Huber's  last  problem-solving  stage,  that  of  monitoring  the 
situation.  The  Hypothesis  element  is  that  aspect  of  intelligence 
that  involves  creating  alternative  hypotheses  to  explain  the 
possible  cause(s)  of  the  problem,  evaluating  the  adequacy  of  each 
hypothesis,  and  selecting  one  or  more  hypotheses  as  the  most 
likely  cause(s)  of  the  data.  It  is  important  to  note  that  more 
than  one  hypothesis  can  be  appropriately  selected  either  because 
of  the  uncertainty  and/or  ambiguity  in  the  data  (Daft  and  Lre.:gel, 
1986),  or  because  there  is  more  than  one  cause  of  the  problem 
(Hammond,  1966).  Regardless,  on  the  basis  of  the  selected 
hypothesis  (or  hypotheses),  the  decision  maker  (and  senior 
associates)  generate  options  for  solving  the  problem.  The  Option 
element  explicitly  differentiates  between  option  creation, 
evaluation,  and  selection  activities.  Finally,  on  the  basis  of 
the  selected  option,  the  decision-maker  (or  decision-making  team) 
takes  action,  which  includes  the  planning,  organization,  and 
execution  of  a  Response  to  the  problem,  analogous  to  the  fourth 
step  in  Huber's  problem  solving  framework.  (Note:  Wohl's 
distinction  between  the  Hypothesis  and  Option  :omponents  is 
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analogous  to  Hogarth's  (1987)  general  distinction  between 
prediction  and  evaluation.) 


The  Option  component  follows  the  Hypothesis  component  in  all 
three  paradigms.  Options  are  not  generated  in  a  vacuum.  Rather, 
they  are  generated  in  response  to  our  hypotheses  with  regard  to 
what  is  happening  to  us,  why  it  is  happening,  and  its 
implications,  though  this  does  not  imply  that  every  option 
generation  situation  requires  a  causal  focus.  Rather,  the 
paradigms  imply  that  in  many  situations  a  causal  focus  is 
essential  to  good  option  generation  and  subsequent  decision 
making.  This  is  clearly  the  case  in  a  military  context. 

According  to  Fitzgerald  and  Grossman  (1987),  "The  real 
objective  of  winning  the  information  war  is  to  enable  a 
decisionmaker ,  from  the  warfare  commander  on  down,  to  perceive 
his  enemy*  s  intentions,  to  assess  his  options  accurately  and  to 
choose  the  optimal  course  of  action"  (underlining  theirs).  The 
purposeful,  goal-oriented  behavior  of  commanders  is  no  different 
from  that  of  executives  or  the  man  in  the  street  (Beach  and 
Mitchell,  1987)  or,  for  that  matter,  the  rat  in  the  runway 
(Tolman  and  Brunswik,  1935).  We  now  turn  to  the  problem-solving 
literature  for  empirical  support  indicating  a  strong  relationship 
between  problem  definition  (i.e.,  situation  assessment)  and 
option  generation,  evaluation,  and  selection. 

3.4.2  Problem  Solving  Research  -  Direct  support  for  the 
premise  that  situation  assessment  affects  option  generation  and 


evaluation  comes  from  the  problem-solving  research  using 
protocol  analysis.  According  to  Newell  and  Simon  (1972), 

"this  pattern  of  symptom-remedy  has  by  now  become  familiar  to  us 
in  a  variety  of  contexts.  It  is  means-ends  analysis,  for  the 
moves  generated  are  relevant  to  'remedying'  the  feature  that 
serves  as  symptom." 

3. 4. 2.1  Task  Characteristics  -  The  typical  procedural 
approach  in  problem-solving  research  is  to  ask  people  to  talk  out 
loud  as  they  attempt  to  solve  a  provided  problem.  Numerous 
different  types  of  problems  have  been  studied  (e.g.,  chess, 
programming,  physics,  logic,  cryptarithmetic)  as  well  as 
alternative  representations  of  the  same  problem  structure,  called 
"problem  isomorphs"  by  Simon  and  Hayes  (1976).  The  reason  for 
having  people  talk  out  loud  is  to  trace  their  problem-solving 
behavior  as  it  occurs;  thus,  it  is  not  susceptible  to  logical 
fallacies  in  recall  or  the  hindsight  bias  (e.g.,  see  Ericsson  and 
Simon,  1984).  The  audio  tapes  are  transcribed  for  subsequent 
analysis,  which  is  why  the  approach  is  often  referred  to  as 
protocol  analysis.  The  typical  protocol  analysis  has  been  used 
to  infer  decision  processes  by  constructing  problem  behavior 
graphs  (Newell  and  Simon,  1972),  which  resemble  flowcharts 
composed  of  nodes  (knowledge  states)  connected  by  arrows  (process 
operations ) . 
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3. 4. 2. 2  Important  Findings  -  Problem-solving  research 
is  vast  and  has  generated  numerous  important  findings.  Of 
importance  here,  however,  is  the  identified  strong  relationship 
between  the  Hypothesis  and  Option  components  in  Wohl  (1981) 
paradigm.  For  example,  in  their  review  and  discussion  of  de 
Groot's  (1965,  1966)  process-tracing  analysis  of  novice  and 
expert  chess  players,  Newell  and  Simon  point  out  that,  "  ... 
during  the  first  moments  —  for  example,  15  seconds  more  or  less 
—  during  which  he  is  exposed  to  a  new  position,  a  skilled  human 
player  does  not  appear  to  engage  in  a  search  of  move  sequences. 
Instead,  he  appears  to  be  occupied  with  perceiving  the  essential 
properties  of  the  position  .  .  .  which  will  suggest  possible 
moves  to  him  and  help  him  to  anticipate  the  consequences.  He 
appears  to  be  gathering  information  about  the  problem,  rather 
than  seeking  an  actual  solution.”  In  short,  the  human  problem- 
solver  is  testing  hypotheses  to  explain  the  data  or  "defining  the 
situation"  (p.  761).  Once  he  has  done  so,  possible  solutions  are 
generated  and  evaluated,  apparently  in  depth.  Again,  to  quote 
Newell  and  Simon,  "humans  playing  chess  spend  much  of  their  time 
searching  in  the  game  tree  for  the  consequences  of  the  moves  they 
are  considering  .  .  .  The  search  is  highly  selective,  attending 
to  only  a  few  of  the  multitude  of  possible  consequences."  People 
do  not  search  for  a  complete  list  of  options,  but  rather,  a 
quality  one  based  on  their  assessment  of  the  situation  and  their 
goals. 

From  the  perspective  of  previous  option  generation  research,  what 
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is  particularly  interesting  about  the  problem-solving  research 
reviewed  by  Newell  and  Simon  (1972)  and  Ericsson  and  Simon  (1984) 
is  the  finding  that  expertise  is  related  to  a  person's  ability  to 
generate  correct  problem  representations,  that  is,  the  correct 
hypotheses  or  causes  for  the  observed  data.  Continuing  with  the 
chess  example,  de  Groot  (1966)  demonstrated  that  chess 
grandmasters  were  significantly  better  able  than  good,  but 
weaker  players  to  reproduce  a  chess  board  after  an  extremely 
brief  exposure  (3-7  seconds).  However,  this  capability  was 
totally  dependent  on  the  meaningfulness  of  the  chess  board;  the 
superiority  vanished  if  the  chess  pieces  were  randomly 
distributed  on  the  board.  Chase  and  Simon  (1973)  replicated  and 
extended  this  finding.  Better  players  are  better  able  to  infer 
causes  (i.e.,  possible  opponent  strategies),  but  only  for 
meaningful  data. 

More  recently,  a  process-tracing  study  by  Isenberg  (1986)  found 
that  action  plan  effectiveness  was  directly  related  to  the  amount 
of  analogical  reasoning  performed  by  12  general  managers  solving 
a  Harvard  Business  School  case.  Bouwman  (1984),  who  analyzed  the 
protocols  for  three  Certified  Public  Accountants  and  five 
graduate  students  evaluating  financial  cases,  found  that  "... 
experts  regularly  summarize  the  results,  and  formulate 
hypotheses.  Such  'reasoning'  phases  further  direct  the  decision 
making  process."  A  longitudinal,  process-tracing  study  by 
Schweiger,  et  al .  (1985),  which  used  the  UCLA  Executive  Decision 
Game  as  the  task,  found  that  subjects  who  engaged  in  causal 


reasoning  performed  significantly  better  than  those  who  did  not. 
In  Cohen's  (1987)  study  of  Air  Force  pilots,  he  found  that 
"the  pilot  who  adopts  a  worst  case  strategy  is  not  really 
suppressing  uncertainty;  he  knows  perfectly  well  that  other 
outcomes  are  possible.  Rather,  he  is  adopting  assumptions  which 
enable  him  to  focus  on  a  concrete,  causally  modeled  state  of 
affairs  as  opposed  to  an  abstract,  non-realizable  average  or 
expected  value.  He  may  subsequently  wish  to  undo  these 
particular  asumptions  and  explore  another  set,  which  implies 
another  concrete,  causally  modeled  state  of  affairs." 

In  sum,  the  problem-solving  literature  strongly  suggests  that  a 
causal  focus  has  a  significant  effect  on  decision  behavior  and 
quality.  Option  generation,  evaluation,  and  selection  is  a 
direct  function  of  situation  assessment.  From  the  perspective  of 
strategic  decision-making,  the  quality  of  operations  is  dependent 
on  understanding  the  inference  processes  in  intelligence. 

Effective  intelligence  is,  in  turn,  dependent  on  understanding 
what  type  of  information  is  important,  and  when  it  needs  to  be 
received,  by  operations  personnel.  (See  Adelman  and  Thompson, 
1989,  for  the  description  of  an  approach  to  measuring  the 
performance  of  tactical  intelligence  units  from  the  users' 
perspective.)  SJT  research  has  shown  that  computer  technology 
can  be  effectively  used  to  improve  the  interpersonal 
understanding  of  cognitive  processes  and,  in  turn,  group 
decision-making . 


3.4.3  Social  Judgment  Theory  -  Considerable  SJT  research 

has  focused  on  interpersonal  learning  where  two  or  more  persons 

work  together  to  perform  a  task.  Figure  3.3  presents  the  "triple 

system  case"  where  one  system  is  the  task  and  the  other  two 

systems  represent  the  cognitive  systems  of  two  cooperating  group 

members.  As  can  be  seen,  the  triple  system  case  (Figure  3.3)  is 

a  natural  extension  of  the  double  system  case  (Figure  3.1).  The 

Lens  Model  Eguation  (LME)  can  be  used  to  assess  the  task 

performance  parameters  of  each  person  and,  more  importantly  here, 

the  agreement  between  them.  That  is,  r  can  be  used  to  represent 

a 

the  overall  level  of  agreement  in  the  judgments  of  the  two 

persons.  G  can  be  used  to  represent  the  similarity  in  the 

judgment  processes  of  the  two  persons.  Rs  and  Rs  can  be  used 

1  2 

to  represent  the  cognitive  control  of  each  person's  judgmental 
process . 

Consequently,  low  agreement  in  a  cooperative  decision-making  task 
can  be  the  result  of  dissimilar  cognitive  process  and/or  low 
cognitive  control.  Dissimilar  cognitive  processes  are  typically 
the  result  of  poor  interpersonal  understanding  of  how  each  person 
is  combining  information  to  make  a  judgment.  Cognitive 
technology  can  enhance  this  understanding,  as  well  as  cognitive 
control,  and  hence  agreement  and  subsequent  performance.  Before 
considering  this  important  finding  in  more  detail,  we  briefly 
overview  the  characteristics  of  SJT  tasks  studying  interpersonal 
understanding  and  agreement. 


3. 4. 3.1  Task  Characteristics  -  As  discussed  earlier 
in  this  review,  cognitive  technology  has  been  used  to  improve 
understanding  and  communication  in  both  the  laboratory  (e.g.,  see 
Hammond  and  Brehmer,  1973)  and  in  real  world  applications  (e.g., 
see  Hammond  and  Adelman,  1976;  Hammond  and  Grassia,  1985).  In 
this  section,  we  first  overview  the  characteristics  of  tasks  used 
in  laboratory  research  and  then,  those  used  in  real  applications. 

Brehmer  (1976)  and  Rohrbaugh  (1988)  have  reviewed  a  host  of 
laboratory  studies  evaluating  the  potential  value  of  cognitive 
technology  to  improving  understanding  and  reducing  conflict.  The 
basic  approach  is  to  first  separately  train  members  of  the 
eventual  decision-making  group  to  have  different  strategies, 
called  "policies,"  for  making  inferences  and/or  decisions.  For 
example.  Person  #1  may  be  trained  to  place  a  high  relative  weight 
and  a  negative  linear  function  on  Cue  A,  but  no  weight  on  Cue  B 
when  making  judgments.  In  contrast.  Person  #2  may  be  trained  to 
put  no  weight  on  Cue  A,  but  a  high  relative  weight  (with  an 
inverted-U  shaped  function  form)  on  Cue  B.  Then,  after  each 
person  is  trained  to  a  level  of  proficiency  in  making  judgments 
their  way,  the  persons  are  brought  together  to  make  group 
judgments . 

In  the  group  task,  the  task  side  of  the  lens  model  is  structured 
so  that  each  person  must  modify  his  judgment  policy  by  learning 
from  the  other  person  in  order  for  the  group  to  perform  well. 
Continuing  with  the  above  example,  the  group  task  may  require 


that  equal  weight  be  placed  on  Cues  A  and  B;  that  a  negative 
linear  function  form  be  used  to  relate  values  on  Cue  A  to  levels 
of  Desirability;  and  that  an  inverted  U-shaped  function  form  be 
used  to  relate  values  on  Cue  B  to  Desirability.  Control  groups 
would  have  to  learn  to  make  group  judgments  by  talking  with  each 
other  and  getting  outcome  feedback;  that  is,  by  the  typical 
approach.  The  cognitive  technology  (or  "cognitive  feedback") 
group  would  be  shown  each  person’s  judgment  policy;  that  is,  they 
would  receive  in  both  pictorial  and  textual  form  a  description  of 
how  each  person  combines  information  to  make  a  judgment. 

The  approach  is  quite  similar  in  real  world  applications,  except 
there  is  no  need  to  train  group  members  to  have  their  own 
judgment  policy;  they  come  with  one.  The  first  step,  therefore, 
is  to  uncover  them.  This  is  accomplished  by  having  the  group 
members  identify  the  cues  that  are  important  to  the  problem. 

Once  consensus  is  reached  on  the  cues,  each  group  member  is  asked 
to  evaluate  a  set  of  cases  comprised  of  different  values  on  the 
cues.  These  cases  may  be  real  ones  (if  there  are  not  excessively 
high  correlations  between  cues)  or  hypothetical  ones  that 
represent  real  cases  if  a  sufficient  number  of  the  latter  are  not 
available.  The  values  on  the  cues  represent  the  values  on  the 
independent  variables  in  a  multiple  regression  equation.  The 
judgments  of  the  group  members  represent  the  dependent  variables. 
Each  group  member ' s  judgments  are  regressed  on  the  independent 
variable  values  in  order  to  obtain  a  "best  fitting"  model  that 
represents  his  judgment  strategy  for  combining  cue  information 


into  a  judgment.  With  the  use  of  software  designed  for  personal 
computers  (Milter  and  Rohrbaugh,  1988),  each  person  can  make 
his  judgments  on-line  and  be  immediately  shown  the  model 
representing  his  judgment  policy.  If  aspects  of  the  model 
(e.g.,  relative  weights,  function  forms,  or  combinational  rules) 
are  found  unacceptable,  changes  can  be  made  on-line.  This  new 
policy  can  then  be  applied  to  the  cases  so  that  the  person  can 
evaluate  its  implications.  The  initial  stage  is  concluded  when 
each  group  member  has  a  model  that  he  thinks  accurately 
represents  his  policy  for  making  judgments. 

The  cognitive  feedback  stage  can  proceed  in  a  number  of  different 
ways,  but  it  typically  follows  the  approach  used  in  the 
laboratory  research.  Group  members  are  first  shown  their 
judgment  policies  so  that  they  can  compare  their  similarities  and 
differences.  Then  they  are  shown  how  the  different  policies 
result  in  different  judgments  for  specific  cases  so  that  group 
members  better  understand  the  implications  of  the  different 
positions.  Third,  the  group  members  try  to  define  a  consensus 
strategy  for  making  future  judgments.  This  group  policy  is  then 
applied  to  the  previously  evaluated  (and  sometimes  new)  cases  so 
that  the  group  members  can  evaluate  its  implications.  Subsequent 
changes  to  the  group  policy  can  be  made  and  the  policy  reapplied 
to  the  cases  until  a  mutually  acceptable  position  is  reached. 

It  is  important  to  note  that  there  are  many  variations  to  the 
above  approach.  For  example,  with  respect  to  strategic 
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intelligence  and  operations,  it  may  be  inappropriate  to  focus  on 
attaining  a  group  policy.  Cognitive  technology  could  still  be 
used,  however,  to  externalize  how  intelligence  and  operations 
personnel  are  combining  information  to  reach  a  judgment. 
Operations  personnel  could  improve  their  understanding  of  how 
intelligence  personnel  are  making  their  judgments  without 
delegating  any  decision-making  authority.  Similarly,  cognitive 
technology  could  be  used  to  externalize  what  information 
operations  personnel  consider  most  important,  and  thereby  help 
guide  the  intelligence  collection  process  without  usurping  the 
managment  responsibilities  of  intelligence  personnel. 

Second,  it  is  important  to  emphasize  that  the  SJT  approach 
represents  only  one  approach  to  modeling  cognitive  processes  for 
the  purposes  of  improving  communication.  In  particular,  decision 
analysis  has  been  extensively  used  in  a  decision  conferencing 
setting  to  facilitate  group  decision  making  (e.g.,  see  Adelman, 
1984;  Weiss  and  Zwahlen,  1982).  Decision  analysis  is  comprised 
of  a  variety  of  different  techniques  for  representing 
individuals'  ir^*»rences  and  valuative  judgments.  Dating  from  its 
application  by  members  of  the  Commander-in-Chief  of  the  European 
Command  (EUCOM),  who  were  faced  with  the  decision  of  whether  or 
not  to  evacuate  US  nationals  from  Lebanon  in  1976  (Kelly, 
Andriole,  and  Daly,  1981),  decision  analysis,  and  the  software 
used  to  support  it,  have  been  effectively  used  to  conduct 
decision  conferences  with  decision  making  groups  for  almost 
fifteen  years  now.  More  recently,  group  decision  support  system 
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technology  has  been  proposed  (e.g.,  see  Andriole,  et  al.,  1990); 
DeSanctis  and  Gallupe,  1987)  as  a  more  general  approach  to 
incorporating  many  different  techniques  for  facilitating  group 
decision  making.  In  addition,  artificial  intelligence  techniques 
have  been  used  to  capture  expertise  in  the  form  of  expert 
systems.  However,  one  thing  that  SJT,  decision  analysis,  group 
decision  support  systems,  and  artificial  intelligence  all  have  in 
common  is  the  importance  of  providing  information  about  cognitive 
processes  ("cognitive  feedback")  as  a  means  of  improving  decision 
making.  As  such,  they  represent  cognitive  engineering  approaches 
to  improved  group  decision-making. 

3. 4. 3. 2  Important  Findings  -  SJT  research  has  led  the 
way  in  demonstrating  the  importance  of  cognitive  feedback  to 
improving  interpersonal  understanding  and  group  decision  making. 
These  findings  are  important  to  consider  when  designing  computer 
technology  to  enhance  group  decision  making.  Reviews  of  the 
laboratory  research  can  be  found  in  Brehmer  (1976)  and  Rohrbaugh 
(1988);  reviews  of  applications  can  be  found  in  Adelman  (1988), 
Hammond  and  Grassia  (1985),  and  Hammond,  et  al.  (1977).  Some  of 
the  important  findings  are  considered,  in  turn. 

First,  interpersonal  understanding  is  often  made  difficult  simply 
by  the  nature  of  people's  judgment  policies.  For  example,  people 
have  great  difficulty  in  learning  another's  policy  if  that  person 
uses  nonlinear  function  forms,  complex  rules  for  combining 
information,  or  has  low  cognitive  control.  In  short,  the  same 
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things  that  make  it  difficult  for  people  to  learn  the  complex 
inference  tasks  represented  by  the  "environmental  system"  in  the 
lens  model,  also  make  it  difficult  for  us  to  learn  about  and 
understand  one  another. 

Second,  discussion  is  an  ineffective  medium  for  interpersonal 
understanding.  First,  we  have  poor  self  in-sight  into  how  we 
make  judgments.  As  Hammond  (1976)  points  out,  "...  verbal 
representations  of  introspective  reports  of  covert  mental 
operations  are  poor  representations  because  of  the  well-known 
inaccuracy  of  introspection;  that  is,  despite  the  best  of 
intentions,  any  introspection  regarding  the  basis  for  a  judgment 
may  simply  be  an  incorrect  description  of  the  cognitive  activity 
that  took  place  .  .  .  [moreover,]  persons  making  the 
introspection  will  not  have  the  conceptual  or  technical  ability 
required  to  describe  their  judgment  processes  completely;  indeed 
they  will  not  even  know  what  a  complete  description  should 
consist  of." 

Second,  language,  for  all  its  beauty  and  versatility,  is  often  an 
ambiguous  medium  for  communication.  As  noted  in  the  review  of 
research  on  cognitive  biases,  Kent  (1964)  and  Moore  (1977)  have 
shown  that  people  attach  very  different  meanings  to  the  same 
expressions  of  uncertainty;  Fischhoff,  et  al.  (1980)  and  others 
have  shown  that  the  answer  one  gets  depends  on  how  one  asks  the 
question.  Moreover,  in  addition  to  being  ambiguous,  Hammond  and 
Boyle  (1971)  note  that  language  is  linear, 


.  .  linear  in  the 


sequence.  Severe  demands  are  thus  made  on  the  learner's  ability 
to  remember  and  to  integrate  sequentially  presented 
relationships.  Consequently,  even  if  a  verbal  description  of 
policy  is  accurate  (which  is  unlikely,  because  of  the  quasi- 
rational  nature  of  policy  judgments;  see  Summers,  Taliaferro  & 
Fletcher,  1970)  the  listener  is  unlikely  to  be  able  to  use 
effectively  the  information  available  to  him  (see  Miller, 

Brehmer,  &  Hammond,  [1971])." 

Third,  ambiguity  in  language  is  confounded  by  inconsistency  in 
judgment.  SJT  research  clearly  shows  that  inconsistency  in 
judgment  (i.e.,  less  than  perfect  cognitive  control)  is  the 
typical  state  of  affairs.  Even  if  one  can  perfectly  describe 
one's  judgment  policy,  one  cannot  perfectly  implement  it. 

Focusing  on  outcome  feedback  (i.e.,  the  other's  judgments)  versus 
cognitive  feedback  (i.e.,  the  other's  policy)  is  an  ineffective 
form  of  learning.  Moreover,  inconsistency  between  the  judgments 
one  makes  and  the  descriptions  one  gives  often  causes  confusion 
and  sometimes  distrust.  The  problem  is  potentially  compounded  by 
logical  fallacies  in  recall  and  the  fundamental  attribution  bias 
of  focusing  on  the  person  rather  than  the  situation. 

The  third  major  SJT  finding  is  that  cognitive  feedback  improves 
interpersonal  understanding  and  group  decision-making.  In 
particular,  using  computer  technology  to  communicate  cognitive 
processes  addresses  each  of  the  above  three  limitations  with 
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language.  First,  by  presenting  a  model  to  represent  one's 
judgment  policy  one  overcomes  the  lack  of  self  insight.  One  can 
explicitly  show  people  the  relative  importance  they  placed  on  the 
different  pieces  of  information,  the  function  forms  relating 
values  on  each  cue  to  their  judgments,  the  combination  rule  that 
appeared  to  best  predict  their  judgments,  as  well  as  measures  of 
cognitive  control.  Second,  this  information  can  be  presented 
pictorially;  it  does  not  have  to  (although  it  can)  be  presented 
mathematically.  Different  sizes,  colors,  graphs,  etc.  can  be 
used  to  represent  different  cognitive  processes.  Third,  these 
pictorial  representations  can  be  used  to  represent  similarities 
and  differences  in  the  judgment  policies  of  different  persons. 

For  illustrative  purposes.  Figures  3.4  and  3.5  show  how 
cognitive  feedback  was  pictorially  presented  to  union  and 
managment  negotiators  in  Balke,  et  al.  (1972).  Specifically, 
Figure  3.4  shows  how  the  width  of  the  lines  in  the  lens  model  was 
used  to  represent  the  relative  importance  each  negotiator  placed 
on  the  four  cues  in  the  contract  dispute.  In  addition,  whether 
the  lines  were  solid  or  hollow  was  used  to  represent  positive  and 
negative  linear  function  forms,  respectively.  Figure  3.5  shows 
how  graphics  were  used  to  represent  differences  in  the  function 
forms  used  by  the  negotiators.  The  representation  of  nonlinear 
functions  is  particularly  important  since  these  are  so  difficult 
to  learn  with  just  discussion  and  outcome  feedback. 

In  closing  this  section  it  is  important  to  emphasize  the  vast 
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Figure  3.5;  Function  Forms/Factors  Differences 


cognitive  engineering  potential  that  now  exists  for  developing 
computer  systems  that  can  enhance  the  interpersonal  understanding 
of  cognitive  processes  and,  thereby,  facilitate  decision-making. 
Our  understanding  of  how  people  make  decisions  and  the  analytical 
methods  for  supporting  them  have  expanded  greatly  within  the  last 
ten  to  twenty  years.  At  the  same  time,  the  capabilities  of 
computer  technology  have,  it  seems,  exploded  while  the  costs  have 
decreased.  These  capabilities  now  include  hypermedia, 
multimedia,  simulation,  animation,  cost-effective  color,  sound, 
multi-dimensionality,  and  true  multi-tasking,  among  others.  The 
task  before  us  is  to  now  learn  how  to  tailor  the  use  of  these 
capabilities  so  as  to  enhance  how  people  think,  communicate, 
decide,  and  act. 
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4.0  ADVANCED  INFOEmATION  TECHNOLOGY 


4.1  Emerging  Tools ,  Techniques  and  Models 

The  field  of  information  technology  —  broadly  defined  —  is 
evolving  at  a  feverish  pace.  "New"  technology  can  become 
obsolete  in  a  few  short  years.  The  good  news  is  that  advanced 
information  technology  is  providing  a  whole  host  of  new  applied 
opportunities.  System  concepts  that  were  once  thought  to  be 
beyond  technical  or  financial  reach  are  now  realistic  options. 
User  requirements  that  call  for  animation,  simulation,  color  and 
multimedia  technology  can  often  now  be  cost-effectively 
satisfied . 

The  hardv/are  and  software  communities  are  providing  newer  and 
more  powerful  systems  every  quarter.  The  challenge  to  system 
designers  is  to  assess  this  progress  and  select  from  the  set  of 
options  those  tools,  models  and  techniques  that  fit  the  problem 
in  question.  Faster  processors  only  make  sense  when  speed  is 
necessary.  Elaborate  interfaces  should  only  be  used  when  users 
require  such  communication  and  when  the  substantive  domain  is 
well  served  by  the  interface  strategy.  There  is  an  important 
theme  here:  if  a  system  will  be  used  for  word  processing 
documents  of  fifty  or  less  pages  then  Intel  80386  or  Motorola 
68030-based  architectures  are  unnecessary.  The  distribution  of 
computing  power  should  correspond  closely  with  intended  and 
anticipated  system  use;  our  use  of  advanced  information 
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technology  should  be  appropriate  and  cost-ef f ecti'’p . 

This  section  of  the  report  examines  some  information  technology 
that  holds  great  promise  for  the  design  and  development  of 
interactive  systems  intended  to  support  analytical  problem¬ 
solving.  The  specific  technology  discussed  here  includes; 

•  Hypertext; 

•  Multimedia; 

•  Animation; 

•  Simulation; 

•  Interactive  (color)  graphics; 

•  Deductive  inferential  knowledge  bases;  and 

•  Adaptive  and  direct  manipulation  interfaces. 

As  suggested  in  Section  1.0,  we  believe  that  enormous  leverage 
can  be  gained  via  the  cognitive  engineering  of  these  tools  and 
that  —  by  and  large  —  they  have  been  grossly  under-exploited  by 
modern  systems  engineers. 

4 . 2  Hypertext 

Hypertext  is  several  things  to  many  people.  Nelson  (1987) 
defines  it  as  "...  a  combination  of  natural  language  text  with 
the  computer's  capacity  for  interactive  branching,  or  dynamic 
display  ...  of  non-linear  text."  Yankelovich,  et  al.  (1985) 
believe  that  hypertext  permits  "...  authors  or  groups  of 


authors  to  link  information  together,  create  paths  through  a 
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corpus  of  related  material,  annotate  existing  texts,  and  create 
notes  that  point  readers  to  either  bibliographic  data  or  the  body 
of  the  referenced  text";  while  Casabianca  (1988)  sees  it  simply 
as  "non-sequential  reading  and  writing." 

Hypertext  is  a  software  incarnated  concept  that  permits  near 
free-form  browsing  though  complex  data  and  knowledge  bases.  The 
simplest  example  is  keyword  searching  through  mounds  of 
information  about,  say,  the  US  Civil  War.  Keywords  —  identified 
by  the  user  via  some  direct  manipulation  of  text  embedded  objects 
like  Lincoln,  Grant  and  Lee  —  would  lead  the  user  directly  to 
those  pages  and  paragraphs  that  contained  information  about  the 
individual  men  (or  user  specified  combinations  of  the  three). 

This  kind  of  searching  requires  the  data  or  knowledge  base  to  be 
arranged  in  some  sort  of  order.  Frisse  (1988a)  suggests  how 
simple  structures,  sequences,  and  hierarchies  can  be  used  to 
organize  data,  information  and  knowledge.  Figure  4.1  (from 
Frisse  Cl988a])  illustrates  the  various  relationships. 

Figure  4.1  and  the  above  definitions  all  suggest  that  hypertext 
is  a  management  system  that  "...  lets  you  connect  .  .  . 
information  using  associative  links"  (Fiderio,  1988).  A  related 
perspective  sees  hypertext  as  an  "  .  .  .  approach  to  information 
management  in  which  data  is  stored  in  a  network  of  nodes 
connected  by  links"  (Smith  and  Weiss,  1988).  Smith  and  Weiss, 
among  others,  define  the  concept  broadly  where  nodes  "...  can 
contain  text,  graphics,  audio,  video,  as  well  as  source  code  or 
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other  forms  of  data."  In  an  impressive  application  of  the 
(broadly  defined)  technology,  Halasz  (1988)  begins  the  shift  from 
hypertext  to  hypermedia  via  a  set  of  notecards  that  permit 
browsing  through  a  large  data  base  of  NATO  missiles.  Figure  4.2 
illustrates  what  a  notecard  from  the  data  base  looks  like,  while 
Figures  4.3  and  4.4  suggest  how  users  can  browse  through  the 
data  base. 

Hypertext  and  hypermedia  (which  presumes  more  than  textual  data) 
provide  systems  designers  with  the  capability  to  link  users  to 
large  complex  data,  information  and  knowledge  bases.  At  the  most 
basic  level,  hypertext  technology  can  be  used  to  reduce  the 
search  time  through  compendia  and  encyclopedias  (Frisse,  1988b). 
More  sophisticated  use  can  easily  permit  users  to  develop 
relationships,  associations  and  hypotheses  among  objects  and,  as 
a  result,  conduct  complicated  on-line  analyses. 

There  are  currently  a  number  of  vendors  that  provide  hypertext 
(and  hypermedia)  applications  software;  there  are  just  as  many 
that  will  build  a  custom  hypertext-based  front-end  to  your  data 
or  knowledge  base.  The  technology  has  arrived  in  concept  and 
application;  it  is  now  possible  to  implement  hypertext  and 
hypermedia  systems  on  low-end  microcomputers.  In  fact,  Apple 
Computer,  Inc.  has  been  giving  away  its  HyperCard  system  for 
several  years  now. 

Hypertext  and  hypermedia  represent  capabilities  that  can  be 
applied  as  stand-alone  solutions  to  data  management  problems  or 
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Figure  4,3:  Hypertext  Notecard: 


Figure  4.4:  Hypertext  Notecard: 


as  embedded  solutions  to  larger  analytical  problems.  Hypertext 
is  a  generic  technology  that  can  be  applied  to  a  variety  of 
problems.  It  will  change  the  way  we  "read"  and  think  about 
reading.  It  will  change  the  way  we  design  management  information 
and  decision  support  systems.  It  will  also  reduce  our 
apprehension  about  the  design  of  user-computer  interfaces  between 
inexperienced  browsers  and  huge  data  bases.  Like  the  other 
technologies  discussed  in  this  section  of  the  report,  hypertext 
represents  a  generic  solution  to  a  variety  of  problems  that  must 
be  defined  as  amenable  to  the  characteristics  of  hypertext  and 
hypermedia.  The  research  reported  here  argues  that  "amenable" 
can  be  defined  via  realistic  interpretations  of  findings  in  the 
cognitive  and  related  sciences. 

4 . 3  Multimedia 

As  suggested  above,  definitions  abound  new  information 
technology  areas.  Multimedia  technology  is  understood  here  as 
the  associative  organization  of  multiple  media,  such  as  text, 
graphics,  sound,  and  video.  Aiken  (1989)  presents  a  framework 
for  envisioning  the  range  of  multimedia  possibilities  (see  Figure 
4.5).  Multimedia  thus  represents  the  marriage  between  hypertext 
and  hypermedia,  that  is,  the  organization  of  data,  information, 
and  knowledge  in  a  variety  of  forms  accessible  in  near  free-form 
fashion. 

Five  years  ago  multimedia  technology  was  expensive  and 
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Figure  4.5:  Aiken’s  Multimedia  Concept 


unreliable.  More  importantly,  there  were  few  application 
concepts.  Early  video  disk  research  mapped  a  good  deal  of  the 
interactive  video  terrain,  but  until  recently  very  few  designers 
considered  how  video,  text,  graphics  and  sound  could  be  synchro¬ 
nized  for  user-computer  interaction  or  problem-solving  purposes. 
Without  question,  the  rise  of  multimedia  applications  correlates 
with  the  fall  in  computing  and  storage  costs.  Color,  interactive 
graphic,  sound  and  video  media  are  now  often  cost-effective  as  is 
the  direct  manipulation  interface  technology  that  permits  smooth 
multimedia  access. 

Apple's  HyperCard  system  represents  an  early  multimedia 
environment.  HyperCard  permits  the  integration  of  graphics, 
text,  sound,  video  (via  hooks  to  other  applications  programs  and 
storage  devices  [like  CD-ROM  players])  and  the  non-sequential 
browsing  through  its  data  and  knowledge  bases.  Newer  systems 
like  Silicon  Beach's  Supercard  adds  color  and  some  other 
multimedia  capabilities.  We  are  only  a  few  years  away  from 
the  widespread  application  of  workstations  like  the  one 
envisioned  by  Aiken  (1989)  in  Figure  4.6. 

Multimedia  workstation  concepts  like  Aiken's  will  permit  systems 
designers  to  satisfy  a  larger  range  of  user  requirements  than  has 
been  the  case  with  traditional  architectures.  Systems  engineers 
responsible  for  designing  and  developing  military  planning 
systems,  for  example,  can  exploit  multimedia  technology  by 
providing  geographical,  graphic  and  video  support  to  users. 
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Figure  4.6:  Aiken’s  Workstation  Concept 


support  that  would  permit  users  to  "see"  the  battlefield. 
Multimedia  technology  can  also  ease  the  interaction  between  users 
and  the  systems  intended  to  support  them.  The  use  of  color  icons 
that  symbolize  relationships  among  system  functions  and  contents, 
for  example,  reduces  operator  workload,  just  as  the  real-time 
conversion  of  text  to  graphics  can  accelerate  problem-solving. 

Like  hypertext,  multimedia  is  a  generic  technology  solution,  but 
--  like  hypertext  --  must  be  applied  judiciously. 

4 . 4  Animation  and  Simulation 

Animation  is  a  tool  that  extends  the  concept  of  interaction  to 
include  real-time  movement  of  objects  and  the  representation  of 
dynamic  processes.  Examples  of  animation  that  come  to  mind 
immediately  include  the  projection  of  trends  over  time, 
diagnostics,  and  dynamic  "what  -  if"  analyses. 

Animation  is  movement  technology;  simulation  is  model-based. 
Until  recently,  simulations  calculated  a  variety  of  events  and 
conditions  and  then  displayed  the  results  to  users  eager  to 
interpret  the  data.  It  is  now  possible  to  animate  simulations; 
users  can  watch  the  simulation  compute  and  see  the  results  as 
they  are  converted  to  dynamic  screen  processes. 

There  are  a  number  of  general-purpose  simulation  programs  that 
support  animation.  Stella  (by  High  Performance  Systems)  and 
Extend  (from  Imagine  Thatl)  are  two  general-purpose  simulation 


programs  for  the  Apple  Macintosh.  There  are  also  a  variety  of 
general-purpose  animation  programs  for  the  Macintosh,  including 
Dimensions  (by  Visual  Information),  Super  3D  and  Supercard  (by 
Silicon  Beach  Software),  and  Macromind  Director  (by  Macromind). 

Animation  and  simulation  tools  (as  well  as  animated  simulations) 
permit  designers  to  convert  static  objects  and  processes  into 
aynamic  ones.  Military  planners  can  "see"  routes  as  they  are 
followed  and  obstacles  as  they  appear  along  the  route.  The 
tracking  of  targets  can  be  seen  dynamically  over  time,  as  well  as 
in  response  to  hypothetical  stimuli.  Animation  can  communicate 
complex  concepts  to  users;  animated  simulation  can  inform 
decision-making  and  problem-solving. 

Animation  and  simulation  expand  our  inventory  advanced 
information  technology  for  systems  design  and  development,  but 
their  use  should  be  restricted  to  interface  and  computational 
requirements  that  call  for  their  unique  capabilities. 

4.5  Interactive  ( Color )  Graphics 

In  the  1970s  interactive  graphics  were  slow  and  expensive. 
Consequently,  unless  requirements  clearly  called  for  graphics 
and/or  color,  they  were  not  incorporated  into  the  interface  or  as 
part  of  system  output.  When  microcomputers  became  commonplace  in 
the  early  1980s,  interactive  color  graphics  were  still  relatively 
expensive.  But  when  architectures  become  more  powerful  in  the 
mid-1980B,  color,  graphics  and  the  means  to  manipulate  them 
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became  affordable.  Concurrently,  our  insight  into  the  optimal 
use  of  color  and  graphics  increased  dramatically.  Schmid  (1983), 
Schmid  and  Schmid,  (1979),  Durrett  (1987),  and  Lewell  (1985)  all 
report  progress  in  computer  graphics  technology  as  well  as  how  it 
can  be  effectively  applied.  Cognitive  psychologists  have 
documented  the  impact  of  text-to-graphics  conversion  (Carroll, 
1987),  and  the  designers  of  popular  applications  programs  (like 
word  processors,  spreadsheets,  and  data  base  managers)  have  begun 
to  consistently  exploit  the  use  of  interactive  color  graphics. 

The  use  of  color  and  graphics  has  been  refined  by  the  human 
factors  community  (Ramsey  and  Atwood,  1979;  Smith  and  Hosier, 
1984).  Empirical  analyses  have  yielded  some  guidelines  for  when 
and  how  to  use  color  and  graphics,  given  user  and  task 
requirements  (Breen,  Miller-Jacobs  and  Miller-Jacobs ,  1987). 

Color,  graphics  and  interactive  color  graphics  can  be  used  to 
accelerate  interaction,  communicate  events  and  conditions,  and 
inform  users  about  system  status.  Andriole  (1986a)  experimented 
with  "graphic  equivalence,"  graphic  explanations  and  embedded 
process  modeling  for  enhanced  user -computer  interaction  in  the 
domain  of  tactical  planning.  Figures  4.7,  4.8  and  4.9  illustrate 
how  graphic  displays  can  be  used  to  convert  text  to  graphics 
(Figure  4.7),  to  explain  system  output  to  users  (Figure  4.8),  and 
inform  users  of  completed  and  remaining  tasks  via  a  graphic 
•'process  model"  (Figure  4.9). 

Windows,  icons,  pull-down  and  pop-up  menus,  among  other  graphic 
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Graphic  Explanation  Concept 


Figure  4.9:  Embedded  Process  Modeling 


techniques,  have  become  popular  over  the  past  few  years  for 
several  reasons.  First  and  foremost,  there  are  countless 
hypotheses  about  the  compatibility  between  human  information 
processing  and  the  use  of  windows,  icons,  and  other  graphic 
navigational  processes  and  cues.  They  are  also  cost-effective. 
Finally,  the  industry  has  longed  for  a  common  user  interface  — 
an  interface  standard,  if  you  will  —  that  would  permit  the 
development  of  integrated  packages  and  second  and  third 
generation  application  programs  without  costly  re-training. 

Hayes  and  Baran  (1989)  have  organized  the  range  of  graphic  user 
interfaces  (see  Figure  4.10).  While  it  has  taken  some  time,  the 
MS-DOS  (and  now  0S2)  world  is  finally  catching  up  to  the 
standard-setting  interface  capabilities  of  the  Apple  Macintosh. 

Color  and  graphics  can  separately  or  together  play  valuable  roles 
in  the  systems  design,  display  and  user-computer  interaction 
processes.  When  coupled  with  direct  manipulation  interface 
technology  color  grapnics  can  enhance  human  performance  by 
augmenting  human  information  processing  capabilities. 

4.6  Deductive  Inferential  Knowledge-Based  Expert  Systems 

Since  the  late  1970s  the  artificial  intelligence  (AI)  community 
has  attempted  to  design,  develop,  and  apply  expert  knowledge 
bases  in  a  variety  of  domains  (Andriole,  1986b,  1988;  Andriole 
and  Hopple,  1988).  The  military  has  been  particularly  enamored 
with  the  potential  of  knowledge-based  problem-solving,  though  its 


Figure  4. 10:  HayesVBaran  Range  of  Graphic  User  Interfaces 


expectations  have  rarely  been  satisfied.  It  is  generally 
recognized  today  that  the  range  of  problems  to  which  AI 
technology  is  applicable  is  far  narrower  than  first  believed. 
Well-bounded,  deductive  inference  problems  appear  best  suited  to 
state-of-the-art  knowledge  representation  and  inference-making 
techniques.  Problems  that  require  induction  and  (especially) 
abduction  are  often  too  complex  for  current  expert  systems 
technology. 

There  are  additional  problems.  Expert  systems  design  and 
development  requires  access  to  articulate  experts.  Many  expert 
systems  are  developed  with  one  or  two  "knowledge  czars"  who  are 
accessible  on  irregular  bases  and  who  sometimes  have  questionable 
expert  credentials. 

There  are  any  number  of  problems  that  defy  intelligent  systems 
technology  of  any  kind  (including  the  current  rage,  neural 
networks).  It  is  beyond  the  capability  of  today's  technology  to 
infer  Soviet  intentions  toward  Hungary  in  1995  or  identify  the 
location  and  date  of  the  next  terrorist  attack.  Current  AI 
technology  can,  however,  perform  routine  diagnostic  tasks, 
recommend  maintenance  procedures,  and  plan  in  well-understood  and 
well-bounded  domains. 

Figure  4.11  presents  some  rules  from  an  expert  diagnostic  system 
known  as  MYCIN.  Developed  by  Shortliffe  and  Buchanon  at  Stanford 
University,  MYCIN  diagnoses  bacterial  infections  and  then 
prescribes  medication.  It  is  a  "backward  chaining"  expert  system 


( 1 )  The  site  of  the  culture  is  blood,  and 

(2)  The  identity  of  the  organism  is  not  known,  and 

(3)  The  stain  of  the  organism  is  Gramneg,  and 
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Figure  4. II:  Rules  From  MYCIN 


that  reasons  from  hypotheses  to  evidence  and  then  back  again.  A 
large  percentage  of  the  successful  expert  systems  are  of  the 
MYCIN  type. 

Figure  4.12  presents  some  rules  from  an  expert  system  developed 
by  Decisions  and  Designs,  Inc.  that  helps  with  the  deployment  of 
early  warning  aircraft.  "If  -  then"  rules  are  the  mainstay  of 
many  expert  systems.  When  procedures  can  be  reduced  to  a 
manageable  set  of  validated  rules,  then  prospects  are  good  for 
the  development  of  a  practical  expert  system.  But  when  the 
domain  and  inference-making  process  are  complex  or  in  any  way 
inexplicable,  then  conditions  are  by  no  means  ideal  for  expert 
systems  development. 

While  knowledge-based  systems  technology  represents  a  powerful 
tool,  there  are  clear  limits  to  its  applied  potential.  Rather 
than  rush  to  domains  that  call  for  AI  we  should  filter 
requirements  through  a  large  methods  filter  prior  to  selecting 
just  the  right  tools,  techniques  or  methods  —  as  suggested  by 
Figure  4.13. 

Figure  4.13  implies  that  there  are  a  variety  of  information 
technology-based  analytical  methods  available  to  the  modern 
systems  engineer.  Figure  4.14  from  Hopple  (1986)  identifies 
numerous  qualitative  and  quantitative  methods  anchored  to  a 
greater  or  lesser  extent  in  advanced  information  technology. 
Analytical  methods  —  regardless  of  whether  they  are 
"intelligent"  or  not  —  constitute  important  members  of  our 


A  threat  is  present  but  its  location  is  not  known. 


Figure  4. 1 2:  Rules  For  Early  Warning  Aircraft  Control 


Problem  Assessment 
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Figure  4. 1 3:  System  Requlrements/t:ethods  Matching 


Ij'j 


Figure  4. 1 4:  Hopple's  Analytical  Methods  Taxonomy 


information  technology  inventory.  But  here  too  compatibility 
with  users,  the  tasks  to  be  performed  on  the  system,  and  what  we 
know  about  cognitive  information  processing  will  determine  where 
and  how  the  methods  should  be  applied. 

4.7  Direct  Manipulation  Interfaces  and  Input  Technology 

Nearly  all  of  the  above  information  technologies,  tools,  methods 
and  techniques  assume  direct  manipulation.  The  state-of-the-art 
in  input  technology  provides  users  with  the  capability  to 
identify  and  retrieve  data,  select  and  execute  functions,  and 
observe  system  processes  via  a  variety  of  interaction  routines 
and  input  devices.  The  now  famous  "point  and  click"  process 
implemented  via  mice  and  trackballs  is  direct  manipulation  at  its 
best.  There  are  additional  options  made  possible  by  new  input 
technologies  such  as  touch  screens,  "datagloves , "  voice 
recognition  systems,  and  even  special  purpose  input  devices  for 
the  manipulation  of  three  dimensional  objects  (Williams,  1988; 
Tello,  1988).  When  these  input  technologies  are  combined  with 
human  factors  and  cognitive  science  guidelines  powerful  user- 
computer  interaction  results. 

We  are  today  on  the  threshold  of  a  new  era  in  human-computer 
communication  where  the  communications  bandwidth  will  widen  in 
response  to  the  confluence  of  human  factors,  cognitive  science 
and  advanced  information  technology. 
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4.8  Adaptive  Human-Computer  Interfaces 


Norcio  and  Stanley  (1988),  among  others,  have  recently  examined 
the  potential  for  real-time  human-computer  adaptation.  According 
to  Norcio  and  Stanley  (1988),  an  adaptive  interface  needs  four 
distinct  kinds  of  knowledge: 

•  Knowledge  of  the  user; 

•  Knowledge  of  the  interaction  process; 

•  Knowledge  of  the  task/domain;  and 

•  Knowledge  of  the  system. 

All  of  this  knowledge  is  intended  to  orchestrate  the  interaction 
process  and  permit  the  system  to  adapt  to  user  preferences  and 
on-line  interaction  behavior,  the  demands  of  the  current  task, 
the  nature  of  the  larger  problem  area,  and  its  own  interaction 
and  processing  capabilities.  The  goal  of  the  adaptive  interface 
research  community  is  to  field  systems  that  respond  to  their 
users,  changes  in  the  problem-solving  environment,  and  the  nature 
of  the  domain  during  the  human-computer  interaction  process. 

While  a  variety  of  research  questions  exist  (Greenberg  and 
Witten,  1985),  progress  has  been  made  in  recent  years  (Rouse, 
1981;  Carbonell,  1983);  we  will  no  doubt  see  the  real-time 
adaptation  concept  broaden  considerably  in  research  and  practice 
over  the  next  few  years. 
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4.9  Additional  Information  Technology 

This  review  of  advanced  information  technology  has  deliberately 
focused  on  those  methods,  tools  and  techniques  that  hold  great 
promise  for  enhancing  analytical  support  and  user-computer 
interaction.  We  have  assumed  progress  in  raw  computing  power, 
which  is  rapidly  occurring  in  a  number  of  areas.  Workstation 
technology,  for  example,  has  made  dramatic  strides  in  the  past 
five  years.  Movement  from  the  Intel  8088  to  the  80286  and  80386 
processors  (and  from  the  Motorola  68000  to  68020  and  68030 
processors)  has  provided  "personal  computer"  and  workstation 
users  with  enough  power  to  perform  many  computationally  intensive 
tasks,  tasks  that  were  impossible  when  the  revolution  in 
microcomputing  began.  Operating  systems  have  evolved  with  the 
processor  technology  all  the  way  to  workstation-based  UNIX  multi¬ 
tasking.  Next  generation  architectures  promise  even  greater 
power  and  flexibility.  The  challenge  will  not  involve  exploiting 
80486-based  machines  for  enhanced  word  processing  efficiency; 
instead,  designers  must  turn  their  attention  to  substantive 
requirements  and  how  new  processing  architectures  can  satisfy  a 
growing  range  of  applications. 

Clearly  there  are  tasks  that  call  for  additional  computational 
power.  Environments  rich  in  data,  information  and  knowledge, 
requirements  that  call  for  data  fusion,  and  problems  that  require 
complex  analyses  will  benefit  tremendously  from  the  new 
architectures.  The  contention  here  is  that  this  power  will  be 
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available  and  cost-effective,  and  that  the  real  challenge  lies  in 
the  extent  to  which  we  can  leverage  the  power  against  new  and 
especially  vexing  problems. 

4.10  Opportunities 

All  of  these  technological  "solutions"  are  essentially  generic. 
There  are  computing  situations  that  will  clearly  call  for  some  or 
even  all  of  them,  and  others  hat  will  require  but  one  or  two. 

As  always,  the  key  lies  in  the  matching  of  the  right  technology 
to  the  right  problem. 

This  report  argues  that  the  matching  piocess  can  be  informed  via 
reference  to  research  in  cognitive  psychology  and  cognitive 
science.  Cognitive  research  can  help  with  the  conversion  from 
requirements  to  system  concept  design;  cognitive  research  can 
serve  as  a  filter  through  which  ideas  about  system  operation, 
interaction  and  communication  can  be  passed  —  prior  to 
implementation.  At  the  most  basic  level,  cognitive  research  can 
help  leverage  traditional  human  factors  findings,  but  at  a  much 
higher  level  can  direct  the  use  of  advanced  information 
technology  to  assure  cognitive  compatibility  and  consistency 
between  systems  and  their  users.  This  latter  leverage  was  the 
objective  of  the  research  described  here. 

The  sxmultaneous  revolutions  in  information  technology  and 
cognitive  science  are  fueling  the  design  and  development  of 
systems  likely  to  ennance  user-computer  interaction  and  human 
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problem-solving.  The  number  and  nature  of  applied  opportunities 
is  rising  as  rapidly  as  user  expectations  about  synergistic 
human-computer  behavior.  There  is  every  reason  to  believe  that 
great  progress  can  be  made  so  long  as  we  remain  diligent  about 
the  integrity  of  the  matching  process.  It  serves  no  purpose  to 
rush  information  technology  into  system  concepts  before  it  has 
been  tested  or  made  reasonably  cost-effective;  nor  is  it  prudent 
to  over-interpret  findings  from  cognitive  science.  The  prototype 
systems  described  below  (see  Section  5.0)  were  designed 
cautiously;  at  the  same  time,  they  suggest  what  can  be  achieved 
when  cognitive  science  and  advanced  information  technology  are 
merged . 
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5.0  COGNITIVE  SYSTEMS  ENGINEERING  OF  ADVANCED  INFORMATION 
TECHNOLOGY  FOR  STRATEGIC  AIR  DEFENSE 


The  previous  sections  of  this  report  have  examined  the  air 
defense  domain,  pertinent  findings  from  cognitive  science,  and 
the  range  of  information  technologies  available  to  the  systems 
engineer.  It  is  now  time  to  merge  all  of  the  above  into  several 
working  demonstration  prototypes. 

This  section  distills  Sections  2,0,  3.0  and  4.0  into  a  set  of 
prototyping  guidelines. 

5 . 1  Air  Defense  Intelligence,  Operations  and  Intelligence/ 
Operations  Requirements 

Air  defense  is  part  of  the  larger  strategic  defense  process.  The 
air  defense  process  —  defense  against  atmospheric  threats  —  can 
be  defined  along  an  intelligence/warning/operations  continuum. 
Intelligence  analysts  seek  the  longest  possible  lead  time  for 
warning  analysts  via  monitoring  military,  political  and  economic 
indicators  of  likely  air  attacks  as  well  as  access  to  the 
indicator  monitoring  that  occurs  at  other  levels  in  the  strategic 
defense  process.  Warning  analysts  convert  intelligence  estimates 
into  likelihoods  of  specific  threats  consisting  of  location, 
timing  and  force  structure.  Operations  personnel  assess  the 
environment,  identify  (pre-selected  and  new)  options,  evaluate 
the  options,  and  recommend  responses  to  perceived  threats. 
Operations  personnel  are  "asset  managers"  who  must  make  trade- 
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offs  among  decision  alternatives.  The  intelligence/warning/ 
operations  interface  requirements  communication  and  the  sharing 
of  data,  information,  and  kno.tfledge.  It  also  requires  a  two-way 
understanding  of  the  intelligence  and  operations  environments. 

5 . 2  High  Leverage  Findings  from  Cognitive  Science 

Section  3.0  provides  a  tour  de  force  of  the  cognitive  science 
literature.  The  survey  and  analysis  provides  insight  into  the 
kinds  of  cognitive  processes  that  can  dramatically  affect  human 
information  processing,  decision-making  and  problem-solving.  The 
key  findings  include: 

INFERENCE -MAKING  FOR  INTELLIGENCE  ANALYSIS  &  PRODUCTION 

•  Inference-making  performance  is  task  dependent; 

•  Human  information  processors  (HIPS)  have  poor  insight 
into  their  own  inferential  processes; 

•  "Cognitive  feedback"  can  significantly  improve  task 
performance ; 

•  HIPS  are  susceptible  to  non-linear-based  deception; 

•  HIPS  use  simple  and  complex  "causal"  schemas  to  explain 
behavior  and  make  inferences; 

•  HIPS  use  "plausible  scenario  generation"  to  test 
and  generate  hypotheses; 

•  HIPS  often  reason  from  experiences,  analogies  or 
"cases" ; 

•  HIPS  rely  on  "cues-to-causali ty"  to  explain  current  and 
future  behavior; 

•  HIPS  process  less  information  than  they  think  they  use 
to  make  inferences; 
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•  Experts  are  as  susceptible  to  poor  insight  about  their 
inferential  processes  as  novices; 

•  HIPS  are  unsure  about  how  to  optimize  data,  information 
and  knowledge  in  inference-making; 

•  HIPS  are  prone  to  "cognitive  reductionism"  to  reduce 
workload  and  overall  mental  effort; 

•  The  perception  of  information  is  not  comprehensive,  but 
highly  selective; 

•  HIPS  tend  to  under-emphasize  "base-rate"  information; 

•  HIPS  weight  the  importance  of  information  on  the  basis 
of  its  perceived  causal  meaningfulness  not  based  upon 
its  statistical  diagnosticity ; 

•  HIPS  are  susceptible  to  the  "availability  bias,"  or  the 
tendency  to  recall  recent  or  highly  publicized  events; 

•  HIPS  selectively  perceive  data,  information  and  knowledge 
on  the  basis  of  experience; 

•  HIPS  are  susceptible  to  confirmation  biases  that  filter 
incoming  information  according  to  pre-conceived  ideas 
and  views; 

•  The  way  information  is  presented  often  determines  how 
it  is  perceived; 

•  HIPS  tend  to  be  conservative  inference-makers; 


DECISION-MAKING  FOR  STRATEGIC  OPERATIONS 

•  Human  decision-makers  (HUDMS)  tend  to  simplify  decision 
problems  by  setting  outcome  aspirations; 

•  HUDMS  often  choose  the  first  decision  alternative  that 
satisfies  the  outcome  aspiration( s ) ; 

•  HUDMS  often  simplify  decision  problems  by  only 
considering  a  small  number  of  alternatives; 

•  HUDMS  use  analogies  to  generate  and  compare  options; 

•  HUDMS  weight  criteria  to  rank-order  decision 
alternatives ; 
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•  HUDMS  selectively  perceive  data,  information  and 
knowledge,  focus  on  confirming  (versus  disconf irming ) 
information,  and  tend  to  anchor  their  judgments; 

•  HUCMS  tend  to  attribute  decision  outcomes  to  chance 
OL  the  complexity  of  the  problem  (not  their  own 
decision-making  deficiencies); 


THE  INTELLIGENCE/OPERATIONS  INTERFACE 

•  Analogical  reasoning  can  enhance  action  plan 
effectiveness ; 

•  Strategic  decision-making  quality  is  dependent  upon 
an  understanding  of  the  inference-making/situation 
assessment  orccess,  and  vice  versa; 

•  Cognitive  feedback  can  enhance  option  generation  and 
evaluation  in  individual  and  group  settings;  and 

•  Communication  lies  at  the  heart  of  shared  models 
and  problem-solving. 


These  and  additional  findings  were  leveraged  in  the  design  and 
development  of  the  prototypes.  Our  intention  was  to  extract  a 
set  of  relatively  uncontroversial  findings  from  the  cognitive 
sciences  and  apply  them  to  the  air  defense  domain  via  the 
creative  application  of  information  technology. 


5.3  Advanced  Informat  ion  Technology 

Section  4.0  presents  an  analysis  of  some  of  the  most  promising 
information  technology  available  today >  The  following  general 
technologies  are  reviewed; 

ADVANCED  INFORMATION  TECHNOLOGY 


Ifi/i 


•  Hypertext; 

•  Multimedia; 


•  Animation; 

•  Simulation; 

•  Interactive  (color)  graphics; 

•  Deductive  inferential  knowledge  bases;  and 

•  Adaptive  and  direct  manipulation  interfaces. 

These  technologies  provide  opportunities  to  the  systems  engineer, 
but  only  when  judiciously  applied.  Our  filters  consist  of  the 
air  defense  requirements  and  the  findings  summarized  above. 

5.4  The  Prototyping  Process 

Andriole  (19a9a,  1989b,  1990)  describes  the  prototyping  process 
in  detail.  Prototyping  is  a  euphemism  for  failure.  Designers  of 
analytical  computer-based  systems  realize  that  it  is  virtually 
impossible  to  capture  and  validate  user  requirements  the  first 
time  through  a  domain.  Iteration  is  almost  always  necessary. 
Failed  requirements  analyses  early  in  the  design  process  should 
be  expected.  Prototyping  accepts  the  inherent  recalcitrance  of 
requirements  engineering  and  endorses  the  development  of  working 
models  of  the  system-to-be  to  validate  user  and  system 
requirements  prior  to  making  large  investments  in  software 
engineering.  Boar's  (1984)  basic  prototyping  model  appears  in 
Figure  5.1,  while  Andriole’s  (1989a,  1989b)  prototyping  life 
cycle  appears  in  Figure  5.2. 

Andriole’s  life  cycle  calls  for  prototyping  as  a  means  to 
requirements  validation.  He  proposes  the  use  of  "storyboards"  to 
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Figure  5.1:  Boor's  Prototyping  Model 
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Figure  5.2:  Andriole’s  Prototyping  Life  Cycle 
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demonstrate  system  capabilities  (prior  to  actual  programming). 
Storyboards  are  linked  displays  of  simulated  system  functions  and 
capabilities.  Figures  5.3,  5.4,  and  5.5  present  some  storyboards 
extracted  from  a  tactical  planning  prototype.  Storyboards  are 
reviewed  by  prospective  system  users,  modified,  and  re-developed 
for  additional  review.  Storyboarding  is  fast  and  inexpensive;  it 
is  thus  a  solid  prototyping  strategy  when  requirements  are  fuzzy 
or  when  advanced  system  concepts  comprise  the  design  objective. 

5.5  The  Storyboard  Prototypes 

This  section  describes  the  prototypes  that  were  designed  to 
demonstrate  how  information  technology  can  be  cognitively 
engineered  to  enhance  air  defense  intelligence,  operations,  and 
the  intelligence/operations  interface.  There  are  three 
prototypes:  one  for  intelligence,  one  for  operations,  and  one 

for  the  intelligence/warning/operations  interface  (which  has 
several  embedded  mini  demonstrations). 

5.5.1  The  Air  Defense  Intelligence  Analysis  and  Production 
Prototype  -  This  prototype  was  conceived  after  the  domain  was 
assessed,  after  the  pertinent  findings  from  cognitive  science 
were  studied,  and  after  the  range  of  information  technologies  was 
surveyed.  The  domain  suggested  that  the  system  could  in  fact 
conduct  a  variety  of  analyses  on  its  own.  This  is  not  because 
our  "intelligent  systems  technology"  is  so  advanced  but  because 
many  aspects  of  the  intelligence  problem  are  relatively  well- 
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Figure  5.  J:  Planning  Storyboard 
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Figure  5.4:  Planning  Storyboard: 


bounded.  Cognitive  science  tells  us  that  analysts  are  better  at 
critiquing  strawmen  hypotheses  than  they  are  at  generating  them; 
the  prototype  thus  generates  hypotheses  at  times  for  the  analyst 
to  accept,  modify  or  reject. 

The  menu  structure  reflects  simplicity  —  and  assumes  that 
intelligence  analysts  are  not  necessarily  experienced  computer- 
based  problem-solvers.  The  notion  of  "process  modeling,"  that 
is,  communicating  to  users  what  the  system  can  (and  cannot  do) 
influenced  our  menu  design.  Note  the  relatively  small  number  of 
commands  and  our  decision  to  make  the  commands  stationary.  The 
user  would  not  have  to  learn  such  a  system;  its  capabilities  are 
obvious.  Once  options  are  selected  they  are  highlighted,  so 
users  can  always  know  precisely  where  they  are  in  the  general 
system  "flow,"  and  what  they  are  doing  at  any  particular  moment. 
Function  and  process  commands  appear  along  the  top  of  the  screen, 
while  system  commands  along  the  bottom,  all  as  suggested  in  the 
following  storyboard  display. 

Sub-menus  appear  when  primary  menu  options  are  selected,  as 
suggested  in  the  displays.  Note  also  the  "portal"  to  operations. 
This  assumes  access  to  the  operations  process  and  all  data  and 
knowledge  bases  that  might  be  used  by  operational  planners. 

The  system  concept  calls  for  windows,  split  screens,  and  mixed 
(system/user)  initiative. 

The  prototype  also  makes  extensive  use  of  graphics,  diagrams,  and 
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patt6ms  that  would  explain  routine,  alert  and  warning  situations.  The 
system  asks  the  user  if  he  wants  more  information  about  the  indicators, 
especially  about  their  source  and  reliability.  User  Queries  and  responses 
are  recorded  by  the  system  and  correlated  with  previous  user-system 
interaction.  User  profiles  are  developed  over  time  and  on-line. 


S2 


indicator  information  came  from,  and  whether  or  not  it  is  reliable.  The 
system  records  the  user's  interest  in  the  indicators;  the  user  requests 
information  about  the  models  used  to  generate  the  Intelligence  estimate 
(by  clicking  on  Models). 


icons.  The  interaction  process  is  alternately  "controlled"  by 
the  system  and  the  user.  As  the  next  several  displays  soqgest, 
the  system  is  capable  of  monitoring  the  analytical  behavior  of 
the  user  and  feeding  it  back  to  him  for  inspection.  This 
capability  is  consistent  with  findings  from  cognitive  science 
that  suggest  that  performance  is  enhanced  via  feedback.  The 
modeling  process  is  presented  in  rough  causal  form  because  human 
information  processors  tend  to  organize  problem-solving  (and 
especially  inference-making)  by  applying  causal  schema. 

Analogical  reasoning  is  supported  by  the  system's  matching 
relevant  cases  to  the  current  analysis. 

The  system  also  provides  analysts  with  the  capability  to  conduct 
"if  -  then"  analyses  by  querying  the  system  about  indicator 
values  if,  for  example,  a  warning  was  imminent. 

All  of  these  and  other  capabilities  are  demonstrated  in  the 
intelligence  prototype.  The  full  prototype  appears  in  Appendix 
B.  Textual  descriptions  of  each  display  are  also  provided. 

These  descriptions  suggest  precisely  where  findings  from 
cognitive  science  and  advanced  information  technologies  have  been 
leveraged. 

5.5.2  The  Air  Defense  Operations  Prototype  -  This  prototype 
is  anchored  in  a  specific  scenario  that  might  well  occur  any  time 
at  the  "Top  ROCC"  (NORAD' s  Region  Operations  Control  Center  at 
Elmendorf  AFB  near  Anchorage,  Alaska).  The  scenario  in  the 
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Threat  for  additional  Information. 
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aircraft)  that  comprise  a  large  segment  of  the  Soviet  threat.  The  system 
provides  options  for  Text  and  More?  data.  The  analyst  clicks  on  Text 


CURREfTT^ITUATiOH 


as  a  hypertext  search  capability.  In  the  example,  the  user  inputs  two  key 
words  ("altitude"  and  "range")  and  the  system  highlights  the  places  in  the 
text  where  references  to  the  words/concepts  appear.  Icons  are  used  to 
search  through  the  data  base;  the  user  then  clicks  on  More?. 


the  analyst  with  the  opportunity  to  check  the  likelihoods  against  inference 
models  and  pertinent  analogies.  The  analyst  then  clicks  on  Project 
Estimates. 
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storyboard  begins  with  a  radar  identification  of  unknown  aircraft 
in  the  region. 

The  menu  structure  for  the  operations  prototype  appears  below. 
Like  the  intelligence  prototype,  the  menu  structure  is  simple  and 
stationary.  It  is  also  designed  to  be  operable  by  either  the 
intelligence  or  operations  analyst.  Like  the  intelligence 
prototype  which  provides  a  portal  to  operations,  the  operations 
prototype  permits  access  to  the  intelligence  analysis  and 
production  system.  Windows,  graphics  and  icons  are  also  used  to 
communicate  with  users. 

The  prototype  suggests  that  operations  tasks  can  be  templated, 
and  that  a  healthy  percentage  of  activity  can  be  system 
controlled.  Intercept  tracks,  for  example,  can  be  calculated 
given  projected  Soviet  tracks.  Criteria  for  selecting  among 
competing  plans  can  also  be  presented  to  the  user  for 
examination.  As  the  following  displays  suggest,  the  system 
presents  the  user  with  its  planning  assumptions  and  the  user  is 
free  to  change  the  assumptions.  The  decision-making  process  is 
made  explicit  through  this  iterative  process,  and  the  user  will 
learn  about  the  assumptions  and  processes  embedded  in  the  system 
as  well  as  his  own.  The  system  also  permits  the  testing  of  plans 
and  the  means  by  which  competing  plans  can  be  compared. 

The  complete  operations  storyboard  prototype  appears  in  Appendix 
B.  Textual  descriptions  of  each  of  the  displays  appear  below 
each  storyboard. 


191 


192 


AIR  DEFENSE  OPERATIONS 


194 


TTJ 

#->  c 

0) 

05  0- 

U 

o 

h- 


cn 

(DO 

Q  ro  ^ 

^  <!»'  u_  "o  5 

S  cn  Q  03  ,^, 

O  ro  °  j->  UJ 

2:  t-  to  ro  (-  r- 

■ .  s  2  g,  g  .1 

y  5  s  s  s  15 

S  t;  ^  =  3 

„  fe  =  52  o  5; 

O  ^  C  ^  t-  <1, 

^  C  —  Q  (U  ^ 

s  OQ  4-J  i2  ^ 

U_  o  ^  O 
^  t_  L_  ro  J—  -Q 

C  03  to  c 

.X  03  TO  I  Q  .— 

4-»  C  “  .si  4-* 

CO  C  .  4-»  fO 

=?  uj  ^  P  E 

4-3  03  (1  L.  p 

•—  4-»  to  —  03  b: 

rn  in  f  i~i  O 


•—  4-»  to  —  03  b: 

CO  to  _  (_  Q.  O 

L.  ^  g  <=  C 

C  03  4-3  fc  (_) - 

P  -t::  <  U  03 

w-»  CZ  Q.)  f 


u  <  U  03 

c=  03  4-3  x:  o  Q 

o  fi  0)  4-3  a.  c! 

(J  T3  u  _  c 

03  o  to  -^  -^  ■*-* 

-C  L-  fO  ^  03 

^  -a  ^  cn 

i"  O  03  §  o 

(J  o  g  c 

X  w  ro  h  -2^  C 

s  g  =  g  2  g 

t_  .lx  ^  to  p 

cx  4-»  03  c  £ 

CO  >  to  O  c 

ra  03  zj  ^  r 

•5^  CL  tn 

i-o 

5  g  s  2  i  5 

to  ••“  tO  03  Cl  » 

CO  CO  >  .£ 

x:  03  •—  o  Q.  ^ 

I—  a:  <  o  I  < 


75 

1 

c_ 

a> 

E 

Q. 

S2 

4-) 

o 

T3I 

o 

t 

lo 

cn 

75 

E 

o 

(O 

o 

CO 

CO 

3 

c 

o 

(O 

i2 

E 

cJ 

CO 

o. 

E 

_i _ 1 

<i> 

4-> 

CO 

20 

Oi 

4-» 

cn 

>. 

<u 

o 

4-» 

0) 

CO 

OP 

CO 

o 

a 

w 

L. 

X 

E 

E 

Q. 

cJ 

c 

<i> 

rs 

CO 

CO 

>* 

ZD 

4^ 

(0 

o 

Hi 

u 

JSi 

a 

4-* 

o 

CO 

CO 

a> 

CO 

t 

o 

c 

o 

c 

a> 

CO 

4-> 

(O 

O 

(_ 

(o 

V 

x: 

CO 

C2 

o 

c 

a; 

CO 

c. 

<u 

CO 

c 

CO 

3 

E 

1 

c 

_o 

4-* 

Q. 

x: 

o 

O 

c. 

CO 

4-» 

Hi 

*0 

4->* 

o> 

< 

•o 

c 

19S 


able  to  understand  system  output  If  they  understand  the  model  from  which 
It  came.  The  system  suggests  that  It  processes  Information,  then  looks  to 
a  set  of  correlated  candidate  0PLAN5,  and  then  scrutinizes  them  vis-a-vIs 
a  set  of  normative  criteria.  The  user  clicks  on  Next 
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The  system  permits  the  user  to  change  the  planning  assumption.  He  clicks 
on  Next. 


5.5.3  The  Air  Defense  Intel liqence/Warning/Operatiors 
Interface  Prototypes  -  There  are  several  mini  demonstrations  of 
how  advanced  information  technology  and  cognitive  science  can 
combine  to  connect  the  intelligence  and  operations  processes. 

The  portals  in  the  air  defense  intelligence  and  operations 
storyboards  are  used  to  suggest  how  the  intelligence/operations 
interface  can  be  supported  via  unconventional  systems  design. 

In  the  first  demonstration  the  operations  planner  has  two  plans 
to  consider.  One  will  yield  a  faster  intercept  of  unknown 
aircraft  than  the  other,  but  the  "longer"  one  will  leave  the 
commander  with  greater  flexibility  to  reconfigure  his  forces,  if 
necessary.  The  decision  to  go  with  the  shorter,  but  less  recon- 
figurable,  plan  hinges  on  the  need  to  reconfigure.  But  the  need 
to  reconfigure  is  part  of  the  larger  intelligence  and  warning 
process.  In  the  demonstration  the  operations  planner  leaves  the 
OPLAN  generation  process  and  enters  the  intelligence  analysis 
process  to  determine  what  the  short  and  longer  term  futures  hold 
for  his  region.  What  are  the  estimates  about  Soviet  strategic 
behavior?  What  is  the  likelihood  that  the  current  situation  is 
part  of  a  larger  Soviet  operation?  Such  questions  should  guide 
the  option  selection  process.  The  answers  will  determine  which 
plan  should  be  selected. 

The  figures  below  suggest  how  the  portal  can  be  accessed  and  how 
operational  planners  can  access  the  intelligence  analysis  and 
production  process.  The  complete  mini-storyboard  appears  in 
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workstation.  This  linkage  presumes  electronic  access  to  intelligence  data 
and  the  intelligence  process.  The  planner  is  presented  a  menu  of 
intelligence  analysis  options.  He  decides  to  look  first  at  the  projections 
of  US/USSR  hostility  likelihoods.  He  clicks  on  Project/Estimates. 
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The  intelligence  system  shows  the  planner  the  reason  why  increased 
tension  is  expected ...  he  clicks  on  Check?. 
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the  intelligence  analysis  and  production  system,  as  well  as  the  capability 
to  browse  through  the  inference  models  embedded  in  the  system.  Specific 
queries  can  be  made  quickly  and  resolved  much  faster  than  if  OPS  tasked 
INTEL  to  conduct  additional  analyses. 


Appendix  B. 


Another  deunjiistrauioii  of  how  inttsractive  sysi-ems  can  support  the 
intelligence/operations  interface  suggests  how  operations 
personnel  can  review  old  and  generate  new  contingency  plans.  The 
system  simulates  how  analysts  can  query  the  system  about  old 
plans  and  generate  new  ones;  it  also  suggests  how  the  old  or  new 
plans  can  be  simulated  to  determine  how  well  or  badly  they  might 
do.  In  the  example  the  simulated  outcome  is  poor  so  the  analyst 
decides  to  probe  the  intelligence  aspect  of  the  problem.  This 
demonstration  represents  how  operations  personnel  can  access 
intelligence  data  and  inferential  processes  directly  from  his 
workstation . 

Some  figures  from  the  storyboard  appear  below. 

In  the  third  example,  the  problem  is  reversed.  The  intelligence 
analyst  is  estimating  short-  and  long-term  threat  and  decides  to 
check  into  the  contingency  planning  of  the  operations  personnel 
to  determine  if  it  is  adequate.  This  represents  an  extension  of 
the  role  of  the  air  defense  intelligence  analyst,  an  extension 
that  may  or  may  not  be  welcomed  by  operations  personnel. 

Several  figures  from  this  third  interface  storyboard  appear 
below. 

The  important  aspect  of  these  interface  prototypes  is  linkage. 
They  assume  that  linkage  among  functions  and  personnel  is 
feasible  and  desirable.  The  linkage  also  challenges  conventional 
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on  Long-Term  Strike  &  Support  Aircraft  to  look  at  what  Intelligence 
believes  is  likely.  The  system  presents  him  with  its  best  estimate.  He 
then  clicks  on  Short-Term  Strike  &  Support  Aircraft.  A  variety  of 
'what-if'*  activities  would  be  possible  via  such  interaction. 


The  system  displays  its  most  recent  intelligence.  The  planner  returns  to 
Air  Defense  Operations  Contingency  Planning. 
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The  Intelligence  analyst  clicks  on  Short-Term  Strike  &  Support 
Aircraft  to  detemlne  the  current  Intelligence  situation. 


208 


The  analyst  then  looks  at  the  Short-Term  Attack  Likelihood. 
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Long-Term  Strike  &  Support  Aircraft  Is  selected  and  displayed. 


doctrinal  wisdom  and  suggests  that  our  systems  should  be  designed 
in  some  radically  new  ways. 

All  three  interface  prototypes  appear  in  Appendix  B.' 
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6.0  EVALUATION  AND  SIZING 


The  prototypes  demonstrate  what  is  feasible.  Their  capabilities 
are  anchored  in  assumptions  about  available  information 
technology  and  how  findings  from  the  cognitive  sciences  can  be 
translated  into  user-computer  interaction  and  overall  system 
capabilities.  We  did  not,  however,  formally  evaluate  the 
prototypes  to  determine  if  they  would  indeed  enhance  or  degrade 
human  performance,  though  we  did  undertake  a  "sizing"  exercise 
to  determine  if  the  "throwaway"  prototypes  could  cost-effectively 
evolve  into  "evolutionary"  ones  (Andriole  1989a,  1989b,  Boar, 
1984) . 

6.1  Contributions  to  Human  Performance 

The  research  reported  here  was  designed  to  test  several 
assumptions.  The  prototypes  suggest  that  it  is  possible  to 
leverage  findings  from  the  cognitive  sciences  via  the  application 
of  selected  information  technologies.  There  is  every  reason  to 
believe  that  if  the  findings  and  technologies  have  been  applied 
carefully  then  human  performance  —  in  the  air  defense  intel¬ 
ligence  and  operations  domain  —  will  be  enhanced.  However, 
without  empirical  results  it  is  impossible  to  determine  the 
nature  or  strength  of  the  contribution. 

Any  systematic  evaluation  must  proceed  according  to  some 
established  guidelines  (Adelman,  1990).  Experiments  must  be 
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designed,  measures  (or  criteria)  of  effectiveness  must  be 
developed  and  validated,  and  data  collection  strategies  must  be 
developed  and  tested.  There  are  a  variety  of  interrelated 
hypotheses  that  must  be  identified  and  organized.  The 
experiments  must  control  for  the  use  and  non-use  of  certain 
displays  and  interaction  processes,  and  assess  how  well  the 
findings/technology  marriage  contributes  to  "better"  and  more 
"accurate"  inference-  and  decision-making,  and  intelligence/ 
operations  communication. 

One  approach  to  evaluation  is  anchored  in  multi-attribute  utility 
assessment  (MAUA)  methodology.  As  Adelman  suggests  (1990),  MAUA 
is  a  powerful  and  versatile  systems  evaluation  methodology.  But 
it  is  also  important  to  think  in  terms  of  multi-method,  multi¬ 
faceted  evaluation,  where  several  approaches  are  used  to  measure 
a  range  of  behavioral  phenomena.  Figure  6.1  presents  a  generic 
MAUA  evaluation  structure  comprised  of  criteria  for  systems 
evaluation  (Adelman,  1990).  Such  a  structure  —  along  with 
whatever  additional  methods,  tools  and  criteria  deemed 
appropriate  —  would  comprise  an  overall  evaluation  strategy. 
Before  the  existing  prototypes  are  converted  into  working 
evolutionary  prototypes,  a  set  of  experiments  should  be  conducted 
to  determine  the  high  and  low  payoff  areas. 

6.2  Implementing  the  Prototypes 

Given  that  any  additional  investment  in  the  prototypes  should  be 
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1.0  OSS/User  Interface  2.0  User-OSS/OrganIzatlon  3  0  Organization/Environment 
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Figure  6. 1 :  Adelman’s  MAUA  Evaluation  Model 


informed  by  experimental  results,  it  is  important  to  note  that 
the  throwaway  prototypes  have  in  fact  been  "sized."  Sizing  is  a 
complicated  process  that  requires  assessments  about  the  costs 
connected  with  scaling  up  the  prototypes  (Andriole,  1989a,  1989b, 
1990).  Typically  five  general  areas  define  the  sizing  process; 

•  Data/knowledge  base  specification; 

•  User-computer  interface  specification; 

•  Specification  of  analytical  methods; 

•  Software  engineering  specification;  and 

•  Specification  of  the  hardware  configuration. 

These  areas  are  used  to  challenge  the  prototype's  conversion  or 
evolutionary  robustness.  Is  it  possible  to  really  build  this 
thing?  Can  it  be  done  cost-effectively?  Does  the  necessary 
technology  exist?  Can  we  program  it  in  the  designated  language? 
Can  the  data  and/or  knowledge  bases  be  developed?  These  are  the 
kinds  of  questions  the  sizing  process  seeks  to  answer. 

6.2.1  Data/Knowledqe  Base  Specification  -  The 
prototypes  call  for  deep  data  and  knowledge  bases.  Much  of  the 
necessary  data  exists;  knowledge  bases  for  inference-  and 
decision-making  would  have  to  be  developed.  The  prototypes  call 
primarily  for  deductive  inferential  processing  and,  therefore, 
reasonably  well-bounded  deductive  knowledge  bases.  It  would  be 
possible  to  develop  evolutionary  prototypes  that  accessed  real 
data  and  deductive  knowledge  bases  cost-effectively.  The 
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prototypes  do  not  call  for  inductive  or  abductive  reasoning, 
though  additional  requirements  analyses  might  very  well  require 
such  reasoning. 

6.2.2  User-Computer  Interaction  Specification  -  In 
spite  of  the  detail  and  flexibility  represented  in  the 
prototypes,  all  of  the  interaction  routines  can  be  cost- 
effectively  developed  and  implemented.  We  did  not  select 
interaction  routines,  or  presume  interaction  devices,  that  are 
beyond  the  reach  of  emerging  technology  or  realistic  budgets. 
This  is  especially  true  if  the  target  platform  remains  the  Apple 
Macintosh  SE,  II  or  IIx.  Apple  Computer,  Inc.  and  its  vendors 
have  placed  advanced  interface  technology  high  on  their  list  of 
priorities  —  and  at  the  top  of  that  list  is  multimedia.  Other 
manufacturers  and  vendors  are  following  suit,  but  they  are 
lagging  well  behind  Apple. 

6.2.3  Specification  of  Analytical  Methods  -  Figure 
6.2  from  Hopple  (1986)  suggests  the  range  of  methods,  tools  and 
techniques  available  to  the  modern  systems  engineer;  Figure  6.3 
suggests  the  ones  that  would  support  the  operation  of  the  next 
generation  prototype. 

None  of  the  methods  are  beyond  the  reach  of  today's  capabilities 
as  always,  the  real  leverage  lies  in  how  the  methods  are  matched 
to  system  functions  and  integrated. 
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Figure  6.2:  Hopple's  Analytical  Methods  Taxonomy 
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Figure  6.3:  Methods  For  Strategic  Air  Defense  Intelligence,  Operations  and  Intel llgence/Operatlons 


6.2.4  Software  Engineering  Specification  -  The  system 
concepts  embedded  in  the  prototypes  are  certainly  programmable. 
Our  analysis  of  the  effort  it  would  take  to  convert  the 
prototypes  into  several  languages  (C,  Ada,'  or  via  fourth 
generation  programming)  did  not  reveal  any  special  problems. 

There  are,  however,  several  assumptions  about  the  hardware 
configuration  (see  below)  that  influenced  our  judgments  about 
engineering  the  software. 

6.2.5  Specification  of  the  Hardware  Configuration  -  If 
the  platform  remains  an  Apple  Macintosh  SE,  II  or  IIx,  then  the 
above  analyses  hold.  We  did  not  size  the  prototypes  beyond  the 
Macintosh  environment.  At  the  same  time,  the  Macintosh 
environment  would  have  to  be  customized  to  accommodate  the  kinds 
of  interaction  processes  and  displays  suggested  in  the 
prototypes.  Video  boards,  video  input  and  output  devices,  large 
memory  and  storage  capabilities,  CD-ROM  players,  and  the  like, 
would  all  become  part  of  the  evolutionary  prototyping 
environment.  It  is  estimated  that  the  necessary  configuration 
would  cost  approximately  $15K  (or  approximately  $20K  if  color 
output  was  considered  necessary) .  It  is  important  to  note  that 

a  very  large  part  of  the  hardware  and  software  architecture 
would  be  off-the-shelf. 
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7.0  SUMMARY  AND  CONCLUSIONS 


Our  research  has  attempted  to  merge  several  distinct  fields  of 
inquiry.  We  began  with  assumptions  about  how  findings  from 
cognitive  science  could  be  leveraged  in  systems  design  and 
development  and  how  the  findings  could  be  amplified  via  the 
application  of  selected  information  technology.  We  demonstrated 
the  leverage  in  several  working  prototypes  designed  to  illustrate 
how  strategic  air  defense  intelligence,  operations,  and  the 
intelligence/operations  interface  could  be  supported  by  some 
advanced  system  concepts.  The  research  has  led  to  several 
conclusions  —  and  recommendations  for  further  research. 

1  Findings 

This  report  describes  the  steps  we  took  to  design  and  develop 
several  demonstration  prototypes.  We  began  with  a  requirements 
analysis  of  air  defense  intelligence,  operations,  and  intelli¬ 
gence/warning/operations.  We  followed  this  analysis  with  a 
review,  assessment  and  distillation  of  the  cognitive  science 
literature,  a  process  that  yielded  a  set  of  findings  particularly 
relevant  to  air  defense  processes.  The  distilled  findings 
represent  a  contribution  unto  themselves.  We  then  undertook  a 
review  and  assessment  of  advanced  information  technology, 
especially  as  it  represented  opportunities  for  enhanced  user- 
computer  interaction.  This  review  examined  a  variety  of 
technologies  —  multimedia,  graphics,  animation,  among  others  — 
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to  determine  if  they  were  now  cost-effective  enough  to  apply.  We 
concluded  that  they  are  and  that  when  married  to  the  distilled 
findings  from  cognitive  science  represent  a  powerful  new  solution 
to  some  perennial  man-machine  systems  problems. 

We  converted  the  marriage  into  several  working  demonstration 
prototypes.  These  prototypes  suggest  that  it  is  possible  to 
design  systems  in  some  unique  multidisciplinary  ways,  that  it  is 
possible  to  widen  the  communications  bandwidth  between  analysts 
and  machines  —  and  among  analysts  using  machines. 

We  also  sized  the  prototypes  to  determine  if  they  could  be 
scaled  up  to  evolutionary  status.  Our  analysis  of  data/ 
knowledge,  user-computer  interaction,  analytical  methods, 
software  engineering,  and  hardware  configuration  requirements 
suggested  that  the  prototypes  could  indeed  be  enhanced. 

Our  Phase  I  research  has  thus  yielded  a  number  of  findings, 
feasibility  assessments,  and  suggestions  for  further  research  and 
development . 

7 . 2  Phase  II  Research  Options 

At  the  top  of  the  list  is  evaluation.  The  system  concepts 
embedded  in  the  prototypes  —  while  anchored  in  the  cognitive  and 
information  sciences  —  have  not  been  empirically  tested.  We  thus 
know  virtually  nothing  about  the  extent  to  which  they  might 
contribute  to  —  or  degrade  —  human  performance  in  air  defense. 
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A  series  of  experiments  should  thus  be  conducted  to  determine  the 
nature  and  strength  of  the  relationships  among  system  capabili¬ 
ties,  specific  air  defense  tasks,  and  human  inference-  and 
decision-making,  and  intelligence/warning/operations  performance. 

The  results  of  these  experiments  should  inform  the  design  and 
development  of  a  set  of  evolutionary  prototypes  which  would 
demonstrate  what  the  new  system  concepts  could  do  in  quasi-  and 
fully-operational  environments. 

The  working  prototypes  could  be  developed  on  an  Apple  Macintosh 
II . 
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Storyboard  suggests  that  the  system  can  monitor  the  threat  environment 
and  detect  abnormal  activity  largely  on  its  own  via  embedded  quantitative 
and  qualitative  pattern  recognition/deductive  inferential  models.  The  user 
"clicks"  on  Current  Situation  and  the  system  responds  with  an  overview 
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zones  or  locations).  The  system  also  provides  the  analyst  with  data  on  the 
missile  and  space  threat,  since  the  system  assumes  that  an  integrated 
attack  is  the  most  likely.  If  bombers  or  cruise  missiles  had  been  detected 
they  would  be  located  on  the  map  and  in  the  appropriate  coverage  area(s). 
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patterns  that  would  explain  routine,  alert  and  warning  situations.  The 
system  asks  the  user  if  he  wants  more  information  about  the  indicators, 
especially  about  their  source  and  reliability.  User  queries  and  responses 
are  recorded  by  the  system  and  correlated  with  previous  user-system 
interaction.  User  profiles  are  developed  over  time  and  on-line. 
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indicator  information  came  from,  and  whether  or  not  it  is  reliable.  The 
system  records  the  user's  interest  in  the  indicators;  the  user  requests 
information  about  the  models  used  to  generate  the  intelligence  estimate 
(by  clicking  on  Models). 
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model"  was  developed  from  the  system’s  embedded  models.  They  are 
presented  side-by-side  to  call  attention  to  the  differences  and  stimulate 
additional  user  analysis.  The  models  are  presented  as  "causar/" influence 
diagrams  of  the  situation  assessment  process,  since  analysts  tend  to 
diagnose  problems  causally.  The  user  then  selects  Analogies. 
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aircraft)  that  comprise  a  large  segment  of  the  Soviet  threat.  The  system 
provides  options  for  Text  and  More?  data.  The  analyst  clicks  on  Text. 
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words  ("altitude”  and  "range")  and  the  system  highlights  the  places  in  the 
text  where  references  to  the  words/concepts  appear.  Icons  are  used  to 
search  through  the  data  base;  the  user  then  clicks  on  More?. 
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presents  data  on  Soviet  air  defense  interceptor  aircraft;  the 
on  More?. 


URRENT 


The  system  displays  information  about  the  ranges  of  two  Soviet  long  range 
bombers  to  provide  graphic/geographic  perspective  on  the  Soviet  threat. 
The  user  then  selects  Threat  Location. 
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The  system  provides  locations  of  Soviet  threat;  user  selects  Threat 
Disposition. 
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the  analyst  with  the  opportunity  to  check  the  likelihoods  against  inference 
modelas  and  pertinent  analogies.  The  analyst  then  clicks  on  Project 
Estimates. 
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system  calculates  the  likelihood  of  the  situation  achieving  the 
:es:  routine,  alert  and  warning.  The  user  requests  information 
change  over  time  by  clicking  on  Check?.  The  system  presents  a 
end  of  things  to  come,  at  least  based  on  the  embedded  models. 
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analytical  models,  but  that  the  indicator-based  explanation  can  be  traced 
to  expected  changes  in  political  indicators.  The  user  inspects  the  political 
events  expected  to  trigger  the  change  to  near-alert  status  by  clicking  on 
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The  system  now  provides  the  analyst  with  the  capability  of  testing 
alternative  hypotheses  via  a  deductive  Inferential  process.  After  clicking 
on  Check?,  the  system  presents  the  analyst  with  a  deduction  based  on  the 
warning  assumption. 
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analysts  to  play  “what  -  if  games  with  the  system,  and  call  the  system  s 
knowledge  base  into  question,  when  appropriate.  The  capability  also  helps 
suggest  how  intelligence  collection  assets  should  be  tasked,  if  the  analyst 
is  "looking”  for  something  in  the  threat  environment.  The  user  then  clicks 
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suggesting  that  a  slight  change  in  the  intelligence  estimate  could  trigger  a 
major  operational  adjustment.  The  threshold  should  be  communicated  to 
the  intelligence  analyst.  OPLANS  should  also  be  communicated  to  the 
intelligence  analyst,  so  that  the  Implications  of  the  estimate  can  be 
integrated  into  the  air  defense  intelligence  and  operations  process. 
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Alaska.  Surveillance  has  detected  several  aircraft  in  the  zones  of 
coverage  that  surround  North  America.  The  target  ID  is  designated  as 
"pending,"  consistent  with  ROCC  operations.  The  user  clicks  on  Evaluate 
Afr  Environment  to  get  more  information  about  the  situation. 
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The  system  determines  that  it  is  not  likely  that  it  is  commerical  or  civil¬ 
ian,  and  that  no  IFF  transmission  has  been  received.  It  concludes  that  the 
aircraft  is  unknown.  The  user  then  clicks  on  Red  Tracks. 
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range  of  possible  aircraft.  The  system  animates  the  tracks  and  then  labels 
them.  The  user  then  clicks  on  Assets,  since  it  is  likely  that  a  scramble 
will  occur. 
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an  actual  system  application  of  the  concept,  the  user  would  have  the 
capability  of  searching  through  assets  for  a  specific  set  (like  F-15s  and 
NORAD  transferred  AWACS).  The  user  requests  additional  information 
about  his  assets. 
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This  display  suggests  that  all  radars  and  communications  systems  are 
operable;  the  user  then  clicks  on  Blue  Intercepts. 
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permit  him  to  see  the  same  plan  in  at  least  two  different  "forms."  The 
system  also  permits  the  user  to  Challenge?  the  plan,  which  he  does.  The 
Challenge?  capability  is  designed  to  evoke  alternative  hypotheses  from 
the  user,  and  to  encourage  the  user  to  critique  the  system-generated 
OPLAN. 


PURREHT  OPERATIONS 


ing  its  option  generation/selection  process  to  the  user.  Users  are  better 
able  to  understand  system  output  if  they  understand  the  model  from  which 
it  came.  The  system  suggests  that  it  processes  information,  then  looks  to 
a  set  of  correlated  candidate  OPLANS,  and  then  scrutinizes  them  vis-a-vis 
a  set  of  normative  criteria.  The  user  clicks  on  Next. 
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optimal  OPLANS  given  certain  (pre-specified)  sets  of  conditions.  This 
additional  insight  into  the  system's  decision-making  process  provides 
users  with  a  deep  understanding  of  the  option  generation/selection 
process  and,  therefore,  into  their  own  decision-making  processes  (via 
their  agreement  or  disagreement  with  the  system).  He  clicks  on  Next. 


The  system  permits  the  user  to  change  the  planning  assumption.  He  clicks 
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The  system  permits  the  user  to  change  the  second  planning  assumption.  He 
clicks  on  Next 
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The  system  invites  the  user  to  change  the  third  assumption.  He  then  clicks 

on  Next. 


well  as  each  criterion's  Importance  vIs-a-vIs  the  others.  The  user 
Inspects  the  weights  and  decides  to  change  them  (by  clicking  on  Change 
Weights?).  Here  the  system  Is  providing  Insight  Into  how  the  option 
evaluation  process  was  structured. 
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The  user  clicks  on  Continue. 


EVAUJATE 


365 


system-generated  one.  It  also  presents  the  system-generatea  uflain  r 
alongside,  so  the  user  can  visually  inspect  each  plan  and  the  two  plans 
together.  He  then  selects  the  Execute  option. 
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The  system  animates  OPLAN  51  and  presents  a  net  effect  expressed  in 
terms  of  mission  objectives,  time,  and  costs. 
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plan  he  prefers  oasea  --  accoraing  co  me  net  ouicome  or  me  suTiuiauon  — 
upon  the  trade-off  between  time  and  asset  reconfigurability.  If  the  threat 
is  expected  to  grow  then  reconfigurability  may  become  very  important.  In 
order  to  decide  which  OPLAN  is  "best/'  the  user  needs  more  information, 
that  is,  more  intelligence. 
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s  display  suggests  that  the  Air  Defense  Operations  planner  is  currently 
ting  some  plans  against  an  embedded  simulator  of  intercept  activity. 

)  system  presents  the  results  to  the  user. 
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violations  is  high,  then  there  may  be  good  reason  to  favor  the 
reconfigurable  plan  over  the  "faster"  one.  But  without  information  about 
the  likelihood  of  these  two  events,  the  planner  cannot  make  a  good 
decision  about  which  plan  to  select.  He  clicks  on  To  Intelligence  for 
real-time  insight  into  the  overall  threat  situation. 
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workstation.  This  linkage  presumes  electronic  access  to  intelligence  data 
and  the  intelligence  process.  The  planner  is  presented  a  menu  of 
intelligence  analysis  options.  He  decides  to  look  first  at  the  projections 
of  US/USSR  hostility  likelihoods.  He  clicks  on  Project/Estimates. 
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QUIT  PAUSE  SAVE 
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This  display  presents  the  planner  with  the  intelligence  analyst’s  estimate 
of  the  US/USSR  hostility  level  (expressed  in  terms  of  routine,  alert  and 
warning  behavior).  The  planner  now  sees  the  estimate  of  increased 
tension  about  90  days  out . . .  he  clicks  on  Check?/: 
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The  intelligence  system  shows  the  planner  the  reason  why  increased 
tension  is  expected  ...  he  clicks  on  Check?. 
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EVALUATE 
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tension  likelihood.  The  system  then  presents  the  planner  with  additions 
intelligence  analysis  options,  and  the  planner  decides  to  inspect  the 
models  that  drive  the  inference-making  process.  He  clicks  on  Explain/ 
Models. 


CURRENT  OPERATIONS 
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over  timeliness.  The  operations  planning  system  provided  direct  access  to 
the  intelligence  analysis  and  production  system,  as  well  as  the  capability 
to  browse  through  the  inference  models  embedded  in  the  system.  Specific 
queries  can  be  made  quickly  and  resolved  much  faster  than  if  OPS  tasked 
INTEL  to  conduct  additional  analyses. 
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CONTINGENCY  PLANS 
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new  contingency  plans.  It  Is  also  possible  to  simulate  old  and 
is.  The  user  clicks  on  Generate  New  and  the  system  presents  the 
-  geographic  map  and  Icons  for  the  planner  to  use  to  construct  a 
at  and  plan. 
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All  I^RE  FjE  N  SHO  RER?AiTil  0 


le  planner  clicks  on  Simulate  to  determine  how  well  (or  badly)  the  plan 
ight  work.  The  planner  can  observe  intercepts,  timing,  and  outcomes. 


CONTINGENCY  PLANS 


check  with  intelligence  because  the  outcome  was  far  from  acceptable.  The 
planner  is  searching  for  the  likelihood  of  the  threat  assumed  in  the 
contingency  plan.  He  clicks  on  To  Intelligence. 


structure  and  interaction  process  is  identical  to  “his"  system.  He  clicks 
on  Long-Term  Strike  &  Support  Aircraft  to  look  at  what  intelligence 
believes  is  likely.  The  system  presents  him  with  its  best  estimate.  He 
then  clicks  on  Short-Term  Strike  &  Support  Aircraft.  A  variety  of 
"what-if"  activities  would  be  possible  via  such  interaction. 


Air  Defense  Operations  Contingency  Planning. 
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current  intelligence  situation. 


The  analyst  then  looks  at  the  Short-Term  Attack  Likelihood. 


Long-Term  Strike  &  Support  Aircraft  is  selected  and  displayed. 
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Long-Term  Attack  Likelihood  is  selected  and  displayed  on  a  time-line. 
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The  analyst  then  enters  the  Air  Defense  Operations  system  via  To 
Operations.  Here  the  intelligence  analyst  decides  to  simulate  some  plans 
based  on  "his"  intelligence  estimates. 


ESTIMATES  &  PROJECTIONS  -4 
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The  system  suggests  that  the  plans  are  adequate,  given  current 
intelligence  estimates.  Had  the  plans  been  inadequate  the  intelligence 
analyst  would  inform  Operations  of  potential  problems. 


